Am Neumarkt 😱

#ML#news 1. https://ai.googleblog.com/2021/11/model-ensembles-are-faster-than-you.html 2. Wang X, Kondratyuk D, Christiansen E, Kitani KM, Alon Y, Eban E. Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models. arXiv [cs.CV]. 2020. Available: http://arxiv.org/abs/2012.01988 Most companies probably have several models to solve the same problem. There are model A, model B, even model C. The final result is some kind of aggregation of the three models. Or the models are cascaded like what's shown in the figure. But it takes a lot of computing resources to run the features through the three models. Wang et al shows that ensembles are not more resource demanding than big models with similar performance in CV tasks.

#fun Lol, thank you Mr Lossfunction. But, which sanitizer are you using? https://www.reddit.com/r/learnmachinelearning/comments/qpolnw/data_cleaning_is_so_must/

#DS#news This is a post about Zillow's Zetimate Model. Zillow (https://zillow.com/ ) is an online real-estate marketplace and it is a big player. But last week, Zillow withdrew from the house flipping market and planned to layoff a handful of employees. There are rumors indicating that this action is related to their machine learning based price estimation tool, Zestimate ( https://www.zillow.com/z/zestimate/ ). At a first glance, Zestimate seems fine. Though the metrics shown on the website may not be that convincing, I am sure they've benchmarked more metrics than those shown on the website. There are some discussions on reddit. Anyways, this is not the best story for data scientists. 1. News: https://www.reddit.com/r/MachineLearning/comments/qlilnf/n_zillows_nnbased_zestimate_leads_to_massive/ 2. This is Zestimate: https://www.zillow.com/z/zestimate/ 3. https://www.wired.com/story/zillow-ibuyer-real-estate/

#ML (See also https://bit.ly/3F1Kv2F ) Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between representations of features in neural networks[^Kornblith2019]. CKA is based on the Hilbert-Schmidt Independence Criterion (HSIC). HSIC is defined using the centered kernels of the features to compare[^Gretton2005]. But HSIC is not invariant to isotropic scaling which is required for a similarity metric of representations[^Kornblith2019]. CKA is a normalization of HSIC. The attached figure shows why CKA makes sense. CKA has problems too. Seita et al argues that CKA is a metric based on intuitive tests, i.e., calculate cases that we believe that should be similar and check if the CKA values is consistent with this intuition. Seita et al built a quantitive benchmark[^Seita]. [^Kornblith2019]: http://arxiv.org/abs/1905.00414 [^Gretton2005]: https://link.springer.com/chapter/10.1007%2F11564089_7 [^Seita]: https://bair.berkeley.edu/blog/2021/11/05/similarity/

#DS#fun Looks familiar. https://xkcd.com/2533/

Live stream finished (220 days)

#DS#ML Microsoft created two depositories for Machine Learning and Data Science beginners. They created many sketches. I love this style. https://github.com/microsoft/Data-Science-For-Beginners https://github.com/microsoft/ML-For-Beginners

#ML ( I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/probably-approximately-correct-pac-learning-and-bayesian-view ) The first time I read about PAC was in the book The Nature of Statistical Learning Theory by Vapnik [^Vapnik2000]. PAC is a systematic theory on why learning from data is even feasible [^Valiant1984]. The idea is to quantify the errors when learning from data and we find that is is possible to have infinitesimal error under some certain codnitions, e.g., large datasets. Quote from Guedj [^Guedj2019]: > A PAC inequality states that with an arbitrarily high probability (hence "probably"), the performance (as provided by a loss function) of a learning algorithm is upper-bounded by a term decaying to an optimal value as more data is collected (hence "approximately correct"). Bayesian learning is an very important topic in machine learning. We implement Bayesian rule in the components of learning, e.g., postierior in loss function. There also exists a PAC theory for Bayesian learning that explains why Bayesian algorithms works. Guedj wrote a primer on this topic[^Guedj2019].   [^Vapnik2000]: Vladimir N. Vapnik. The Nature of Statistical Learning Theory. 2000. doi:10.1007/978-1-4757-3264-1 [^Valiant1984]: Valiant LG. A theory of the learnable. Commun ACM. 1984;27: 1134–1142. doi:10.1145/1968.1972 [^Guedj2019]: Guedj B. A Primer on PAC-Bayesian Learning. arXiv [stat.ML]. 2019. Available: http://arxiv.org/abs/1901.05353 [^Bernstein2021]: Bernstein J. Machine learning is just statistics + quantifier reversal. In: jeremybernste [Internet]. [cited 1 Nov 2021]. Available: https://jeremybernste.in/writing/ml-is-just-statistics

#ML https://www.microsoft.com/en-us/research/blog/turing-bletchley-a-universal-image-language-representation-model-by-microsoft/

#ML (I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/how-do-neural-network-generalize ) There are somethings that are quite hard to understand in deep neural networks. One of them is how the network generalizes. [Zhang2016] shows some experiments about the amazing ability of neural networks to learn even completely random datasets. But they can not generalize as the data is random. How to understand generalization? The authors mentioned some theories like VC dimension, Rademacher complexity, and uniform stability. But none of them is good enough. Recently, I found the work of Simon et al [Simon2021]. The authors also wrote a blog about this paper [Simon2021Blog]. The idea is to simplify the problem of generalization by looking at how a neural network approximates a function f. This is approximate vectors in Hilbert space. Thus we are looking at the similarity of the vectors f, and its neural network approximation f'. The similarity of these two vectors is related to the eigenvalues of the so-called “neural tangent kernel” (NTK). Using NTK, they derived an amazingly simple quantity, learnability, which can measure how Hilbert space vectors align with each other, that is, how good the approximation using the neural network is. [Zhang2016]: Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization. arXiv [cs.LG]. 2016. Available: http://arxiv.org/abs/1611.03530 [Simon2021Blog]: Simon J. A First-Principles Theory of NeuralNetwork Generalization. In: The Berkeley Artificial Intelligence Research Blog [Internet]. [cited 26 Oct 2021]. Available: https://bair.berkeley.edu/blog/2021/10/25/eigenlearning/ [Simon2021]: Simon JB, Dickens M, DeWeese MR. Neural Tangent Kernel Eigenvalues Accurately Predict Generalization. arXiv [cs.LG]. 2021. Available: http://arxiv.org/abs/2110.03922

#visualization "Fail" When visualizing data, the units being used have to be specified for any values shown. But the style of the charts is attractive. :) By chungischef Available at: https://www.reddit.com/r/dataisbeautiful/comments/q958if/recreation_of_a_classic_population_density_map/