DataScience4you2me

In R, the logical || (OR) and && (AND) operators are unique in that they are designed only to work with scalar arguments. Typically used in statements like while(iter < 1000 && eps < 0.0001) continue_optimization() the assumption is that the objects on either side (in the example above, iter and eps) are single values (that is, vectors of length 1) — nothing else makes sense for control flow branching like this. If either iter or eps above happened to be vectors with more than one value, R would silently consider only the first elements when making the AND comparison. In an ideal world that should never happen, of course. But R programmers, like all programmers, do make errors, and using non-scalar values with || or && is a good sign that something, somewhere, has gone wrong. And with an experimental feature in the next version of R, you will be able to set an environment variable, _R_CHECK_LENGTH_1_LOGIC2_, to signal a warning or error ...

It seemed like such a simple movie. The Naked Gun (1988) is slapstick comedy through-and-through, but I never would have guessed (h/t Steven O'Grady) how much detail and planning went into the jokes, especially the baseball scene at the end. There's lots of interesting behind-the-scenes info in Sporting News's breakdown of the movie. Even Drebin's bungled National Anthem performance was composed in advance for the scene: That's all from us for this week. Next week, I'm in Australia for the Melbourne R User Group and the eResearch Australiasia conference, so I expect blogging will be a little lighter than usual. Have a great weekend! from Revolutions https://ift.tt/2IRAqcH via IFTTT

The Economist's Big Mac Index (also described on Wikipedia if you're not a subscriber) was created (somewhat tongue-in-cheek) as a measure to compare the purchasing power of money in different countries. Since Big Macs are available just about everywhere in the world, the price of a Big Mac in Sweden — expressed in US dollars — gives an American traveler a sense of how much more expensive things will be in Stockholm. And comparing the price of a Big Mac in several countries converted to a single baseline currency is a measure of how over-valued (or undervalued) those other currencies are compared to that baseline. Since its inception in 1986, the Big Mac Index has been compiled and calculated manually, twice a year. But starting with the most recent published index (July 2018, shown above), the index is now calculated with R. This is the first example of a new program at The Economist to publish the data and methods behind its journalism, and here the data and code behind the B...

I wasn't at the Use of R in Official Statistics (uRos2018) conference in the Netherlands last month, but I'm thankful to Jeroen Ooms for sharing the slides from his keynote presentation. In addition to being a postdoc staffer at ROpenSci, Jeroen maintains the official repository for the daily R builds on Windows — you might recognize his name from the verification certificate that pops up when installing R on Windows. His uRos2018 talk provides a fascinating glimpse into the complex systems, dependencies, and processes that come together to make installing R as easy as as double-click. On thing I found if interest is that while R is a complex program in its own right — made up of a mix of C (32%), Fortran (24%) and R (37%)  code — it also relies on a number of external libraries to perform some of its underlying tasks. For example: When you multiply two matrices or perform other linear algebra operations, R uses the BLAS and LAPACK libraries for the calculation. (Microsoft R o...

Since 2015, the R Consortium has funded projects of benefit to, and proposed by, the R community. Twice a year, the R Consortium Infrastructure Steering Committee reviews grant proposals and makes awards based on merit and funds available. (Those funds come, in turn, from the annual dues paid by R Consortium members.) If you'd like to propose a project of your own, the deadline for submission for the Fall 2018 Call for Proposals is October 31. Not sure what kind of project might be approved? In general, the R Consortium grants fund projects in these categories: Creation of general infrastructure supporting R, like the R-hub project which provides build services for R package developers on every platform supported by R. Development of new R packages, like DBI for R which provides a general interface between R and databases.  Support for working groups to meet, discuss and plan new initiatives in the R community, like Certification for R developers. R community programs, like R ...

Thanks to Mike Loukides I recently discovered IDEA, a series of algorithm explainers presented as IKEA assembly instructions. It's a brilliant concept: IKEA instructions have to be clear and easy to follow, in sequential steps, without use of language. There are entries for determining a convex hull and for various sorting algorithms, but this one on public key encryption is my favorite (not least because it's the first time I feel like I "get" the concept). Credit: idea-instructions.com. License: CC by-nc-sa 4.0 That's all from us for this week. Have a great weekend, and we'll be back next week. from Revolutions https://ift.tt/2OH7UQm via IFTTT

Here are some conferences focused on R taking place in the next few months: Oct 26: Nor'eastR Conference (Providence, RI). A one-day R conference, organized by grassroots R community members in the Northeastern US.  Oct 27: SatRdays Belgrade (Serbia). Another city joins the popular SatRDays series of one-day, community led conferences.  Nov 7: EARL Seattle (Seattle, WA). The EARL London conference goes on a Stateside road trip, with a 1-day stop in Seattle. Nov 8-9: DC R Conference (Washington, DC). From the same organizers as the fantastic New York R Conference, this DC-based outpost will feature speakers including Mara Averick, Roger Peng, and Emily Robinson. Nov 9: EARL Houston (Houston TX). The EARL roadshow continues with a stop in Texas. Nov 13: EARL Boston (Boston, MA). The EARL roadshow concludes with a 1-day event in Boston. Dec 8: SatRDays DC (Washington, DC). Talk proposals for this Washington, DC instance of SatRDays are open until Oct...

In case you missed them, here are some articles from September of particular interest to R users. R code by Barry Rowlingson to replicate an XKCD comic about curve fitting. The rayshader package creates 3-D relief maps in R with perspective, shadows, and depth of field. The R Developer's Guide to Azure, with links to documentation for Azure cloud services integrating R. A review of many commercial applications of R presented at EARL London 2018. Roundup of AI, Machine Learning and Data Science news from September 2018. A Shiny app using the Custom Vision API to identify pictures (or not) of hotdogs. Two academic articles use survey techniques to estimate casualties from Hurricane Maria in Puerto Rico. Yihui Xie describes the benefits of RMarkdown documents in response to criticism of Jupyter Notebooks. A video demonstrates the use of R and Python in Azure HDinsight.  Similarity analyses in R used to identify candidate authors for an anonymous op-ed in the New York Times. A review o...

Microsoft Ignite, Microsoft's annual developer conference, wrapped up last week and many of the big announcements focused on artificial intelligence and machine learning. The keynote presentation from Microsoft's Cloud AI lead Eric Boyd showcases the major developments, or you can check out his accompanying blog post for a written summary.  In this post, I'll dive down into the details of the specific announcements and provide links to the associated recorded presentations from Ignite and other technical documents. Tools for AI developers Azure Machine Learning Service (Azure ML) adds several new capabilities. These include a new Python package that developers can use to prepare data for analysis, automate machine learning and deep learning model training, track model performance under various conditions, and automate the model selection process (see below). It supports a variety of execution environments, from local desktops to GPU clusters, and deploy trained model...

You probably saw this XKCD last week, which brought a grimace of recognition to statisticians everywhere: It's so realistic, that Barry Rowlingson was able to reproduce all but two of the "charts" above with a simple R function (and a little help from the xkcd ggplot2 theme): And now for @revodavid et al, with the xkcd package and font! (still two more to do...) pic.twitter.com/3aVHis23Gl — Barry Rowlingson (@geospacedman) September 21, 2018 You can find the R code behind Barry's reproductions here, and I'm sure he'd welcome contributions for the remaining two charts :).    from Revolutions https://ift.tt/2ImR1ol via IFTTT

Data scientists often work with geographic data that needs to be visualized on a map, and sometimes the maps themselves are the data. The data is often located in two-dimensional space (latitude and longitude), but for some applications we have a third dimension as well: elevation. We could represent the elevations using contours, color, or 3-D perspective, but with the new rayshader package for R by Tyler Morgan-Wall, it's easy to visualize such maps as 3-D relief maps complete with shadows, perspective and depth of field: 👌 Dead-simple 3D surface plotting in the next version of rayshader! Apply your hillshade (or any image) to a 3D surface map. Video preview with rayshader's built-in palettes. #rstatsCode:elmat %>% sphere_shade() %>% add_shadow(ray_shade(elmat)) %>% plot_3d(elmat) pic.twitter.com/FCKQ9OSKpj — Tyler Morgan-Wall (@tylermorganwall) July 2, 2018 Tyler describes the rayshader package in a gorgeous blog post: his goal was to generate 3-D representations o...

If you want to run R in the cloud, you can of course run it in a virtual machine in the cloud provider of your choice. And you can do that in Azure too. But Azure provides seven dedicated services that provide the ability to run R code, and you can learn all about them in the new R Developer's Guide to Azure at Microsoft Docs. The services include: Data Science Virtual Machine: a customized VM to use as a data science workstation or as a custom compute target. This includes Microsoft R Open, Microsoft Machine Learning Server, and Microsoft ML Services in SQL Server, which can also be run locally. ML Services on HDInsight: a cluster-based system for running R analyses on large datasets across many nodes. Azure Databricks: a collaborative Spark environment that supports R and other languages. Azure Machine Learning Studio: a drag-and-drop app for Azure's machine learning experiments, that supports custom R scripts. Azure Batch: a variety of options for economically run...

I was about the same age as student pilot Maggie Taraska when I had my first solo flight. Unlike Maggie, I didn't have to deal with a busy airspace, or air traffic control, or engines (I was in a glider), or — most significantly — one of my landing wheels falling off during the flight. But Maggie handled the entire situation much more coolly than I remember my own short, uneventual, first solo. Just listen to the radio chatter below. If you're in a rush, you can skip the period from 2:00 – 7:30 while she circles the Beverly (MA) airport waiting for a couple of other planes to land first, but the whole thing is worth a listen to hear how she successfully lands with only one wheel on the left side.   That's all from us at the blog for this week. Have a great weekend, and we'll be back next week. from Revolutions https://ift.tt/2I2qe0l via IFTTT

During the EARL (Enterprise Applications of the R Language) conference in London last week, the organizers asked me how I thought the conference had changed over the years. (This is the conference's fifth year, and I'd been to each one.) My response was that it reflected the increasing maturity of R in the enterprise. The early years featured many presentations that were about using R in research and the challenges (both technical and procedural) for integrating that research into the day-to-day processes of the business. This year though, just about every presentation was about R in production, as a mainstream part of the operational infrastructure for analytics.  That theme began in earnest with Garret Grolemund's keynote presentation on the consequences of scientific research that can't be replicated independently. (This slide, based on this 2016 JAMA paper was an eye-opener for me.) The R language has been at the forefront of providing the necessary too...

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications from Microsoft and elsewhere that I've noted over the past month or so. Open Source AI, ML & Data Science News ONNX 1.3 released. The standard for representing predictive models adds a cross-platform API for executing graphs on optimized backends and hardware. A new foundation supporting the development of scikit-learn, the Python machine learning library. Google open sources "dopamine", a Tensorflow-based framework for reinforcement learning research. Google adds a "What-If Tool" to Tensorboard, to interactively explore the robustness and algorithmic fairness of machine learning models. Industry News Google introduces Dataset Search, an index of public datasets from the environmental and social sciences, government, and journalism. The 2018.3 update to the Alteryx a...

I had a great time at the EARL Conference in London last week, and as always came away invigorated by all of the applications of R that were presented there. I'll do a full writeup of the conference later this week, but in the meantime I wanted to share the materials from my own presentation there, "Not Hotdog: Image Recognition with R and the Custom Vision API". I've embedded the slides below: This is an embedded Microsoft Office presentation, powered by Office Online. In the presentation, I showed how easy it is to build a Shiny app to perform image recognition with the Custom Vision API in Azure. I've provided all of the R code behind the app in this GitHub repository, and thanks to the httr package it takes just a few lines of R code to interface with the API to upload images, train a transfer learning model, and classify new images. It's also handy to be able to use the customvision.ai web portal to review your tagged images and model performance.  This w...

With Hurricane Florence battering the US and Typhoon Manghkut bearing down on the Philippines, it's a good time to take a look at the art of visualizing predicted hurricane paths. (By the way, did you know that "typhoon", "hurricane" and "cyclone" are just different names for the same weather phenomenon?) Flowing Data has a good overview of the ways media have been visualizing the predicted path (hat tip: reader MB), including this animation from Axios which does a good job of demonstrating the uncertainty in the forecast: A good thing to be aware of, though, is that the cones around the predicted tracks do not represent the size of the storm, but rather the uncertainty in the position of the center of the storm. For a "live" view though, the place I like to look at is the global wind visualization from the Climate Literacy and Energy Awareness Network. Here's how Florence looks at this writing (3:45AM East Coast time). Click the image to...

President Trump is once again causing distress by downplaying the number of deaths caused by Hurricane Maria's devastation of Puerto Rico last year. Official estimates initially put the death toll at 15 before raising it to 64 months later, but it was clear even then that those numbers were absurdly low. The government of Puerto Rico commissioned an official report from the Millikan Institute of Public Health at George Washington University (GWU) to obtain a more accurate estimate, and with its interim publication official toll stands at 2,975.  Why were the initial estimates so low? I read the interim GWU report to find out. The report itself is clearly written, quite detailed, and composed by an expert team of social and medical scientists, demographers, epidemiologists and biostatisticians, and I find its analysis and conclusions compelling. (Sadly however the code and data behind the analysis have not yet been released; hopefully they will become available w...

Author and research engineer Joel Grus kicked off an important conversation about Jupyter Notebooks in his recent presentation at JupyterCon:  There's no video yet available of Joel's talk, but you can guess the theme of that opening slide, and walking through the slides conveys the message well, I think. Yuhui Xie, author and creator of the rmarkdown package, provides a detailed summary and response to Joel's talk, where he lists Joel's main critiques of Notebooks:  Hidden state and out-of-order execution Notebooks are difficult for beginners Notebooks encourage bad habits Notebooks discourage modularity and testing Jupyter’s autocomplete, linting, and way of looking up the help are awkward Notebooks encourage bad processes Notebooks hinder reproducible + extensible science Notebooks make it hard to copy and paste into Slack/Github issues Errors will always halt execution Notebooks make it easy to teach poorly Notebooks make it hard to teach well  Yihui suggests that m...

Azure HDInisght was recently updated with version 9.3 of ML Services in HDInsight, which provides integration with R and Python. In particular, it makes it possible to run R and Python within HDInsight's managed Spark instance. The integration provides:   R and Python support, with interaction via Visual Studio, VS Code, or RStudio Specialized distributed analytics libraries for R and Python Simple operationalization of R and Python with Spark Interoperability with Sparklyr, pyspark and H2O.ai In the episode of Data Exposed embedded below, you can learn more about the ML Service 9.3 in HDInsight and see some demos of the R and Python integration. Channel 9: Introducing ML Services 9.3 in Azure HDInsight from Revolutions https://ift.tt/2CMZd2e via IFTTT