Amazon's AWS cloud service on Wednesday introduced a new application using generative AI, called HealthScribe, which will transcribe doctors' conversations with patients, and provide an audit trail of the audio used to generate the text.
Amazing announced HealthScribe during its New York City developer forum, AWS Summit New York.
HealthScribe builds on top of AWS's hub for generative AI, called Bedrock, which the company introduced earlier this year. Bedrock is a managed service that provides access to "foundation models," generative AI programs that can be tuned to a variety of use cases, including Amazon's own Titan language model.
Also: Amazon hones in on generative AI at AWS Summit and unveils new AI projects
Amazon said the point of HealthScribe is to relieve physicians of the burden of manually entering notes into an electronic health record.
"By integrating AWS HealthScribe into a clinical application, healthcare providers can leverage built-in text-to-speech capabilities to create robust conversation transcripts that identify speaker roles and segment transcripts into categories (e.g., small talk, subjective comments, or objective comments) based on clinical relevance," said Amazon.
"The application can then use AWS HealthScribe's NLP [natural language processing] and generative AI capabilities to extract structured medical terms, such as medical conditions and medications, and generate discussion-based notes that include relevant details (e.g., key takeaways, reason for visit, and history of the present illness) that a clinician can review and finalize in their EHR."
Also: MedPerf aims to speed medical AI while keeping data private
Noting the possibility of "hallucinations," when a large language model invents things, the announcement points out that "every sentence used in the AI-generated clinical notes comes with references to the original doctor-patient conversation transcripts, allowing clinicians to easily view the historical context of notes for greater accuracy and transparency."
The software is HIPAA-eligible, said Amazon.
In addition to HealthScribe, Amazon on Wednesday announced updates to Bedrock's foundation model hosting. In addition to Titan, customers can now choose from among Stability.ai's Stable Diffusion model for image generation; Claude 2 from Anthropic; and Command and Embed from Cohere, which is a model that will "follow user commands and be useful instantly in practical business applications such as summarization, copywriting, dialogue, extraction, and question answering," said Amazon.
Also: Google's MedPaLM emphasizes human clinicians in medical AI
For more on the announcements at Summit, read Sabrina Ortiz's roundup.
HealthScribe is currently in preview mode, and AWS is offering 300 minutes of free usage. You can read more details on the HealthScribe site.
Artificial intelligence has been running in the background of many of our favorite pieces of technology for years. However, the AI boom ChatGPT brought on shined a spotlight on AI, and it may not have been for the best.
Steven's Institute of Technology released its annual TechPulse report, which measures the public's attitudes toward different technologies. The findings in the 2023 report show that AI's increased popularity may have actually harmed the public's perception of AI.
Also: Adobe Generative Expand adds to Photoshop's AI arsenal
Out of the 2,200 adults surveyed, only 38% said that the positive impact of AI would outweigh the risks, compared to the 48% of respondents who said it would in 2021. In two years, optimism went down 10%.
This negative impression of the impact of AI spread across the different potential applications of AI. The respondents were less likely to say that AI could positively impact personal safety, national security, and personal privacy than in 2021.
Furthermore, 25% of the respondents shared that concern was the emotion they most strongly felt regarding AI.
Also: Generative AI is coming for your job. Here are 4 reasons to get excited
These feelings likely flow from all of the unforeseen, negative situations that have resulted from the widespread use of relatively new technology.
A few of these include ChatGPT's March 20 data leak, FTC investigation of OpenAI's technology, several lawsuits, and even warnings from AI experts about the risks of AI, not to mention AI-driven stock drops and hiring pauses.
With organizations now prioritizing artificial intelligence (AI) and cybersecurity, personal work devices should be built to support these requirements.
Work laptops, for instance, should have firmware that is robust and secure, said Nima Baiati, Lenovo's global executive director and general manager of commercial security. Noting that the firmware or BIOS could pose a significant threat if vulnerabilities are left unaddressed, Baiati said it is a critical component that Lenovo works to continuously enhance.
Also:ChatGPT and the new AI are wreaking havoc on cybersecurity
To do that, the hardware manufacturer is turning to AI and its expertise in supply chain management to address some of these challenges, he told ZDNET. New features from this focus will be rolled out over the next one to two years, though, he declined to divulge details on what these may be.
Instead, he pointed to a cybersecurity innovation center Lenovo opened in Israel in February, which would help the vendor explore how AI can be tapped in areas not traditionally used from a security standpoint.
And with generative AI potentially introducing new security risks, including lowering the threshold for cybercriminals to launch attacks, organizations will need solutions to safeguard against such threats.
Also:These experts are racing to protect AI from hackers
Work devices also will need to provide more compute power, especially with the growing use of cloud-based AI applications, he said. Organizations increasingly are opting to build their own cloud infrastructures to support these processes, so they can gain scale, flexibility, and security. This, however, can significantly drive up spending.
To help mitigate its costs, Lenovo is looking at how to improve efficiencies, such as compute power. Ensuring low latency is critical when some of the AI processing, as adoption grows, will have to be done on the device itself, rather than on the cloud, Baiati noted.
As it is, machine learning, AI, and analytics are among the most urgent priorities for businesses.
Some 43% of CIOs expressed "urgent pressure" to address AI and machine learning as a priority, just below 51% who pointed to cybersecurity, according to a Lenovo report. The survey polled 682 CIOs across nine global markets including China, Singapore, India, the UK, Germany, and the US.
AI and machine learning, as well as data management and analytics, were the top priorities for 43% of the CIOs in China. Some 34% pointed to cybersecurity as their most urgent priority.
Also:The best laptops right now
In India, security was a top priority for 62% of CIOs, as well as for 51% of their counterparts in Singapore. AI and machine learning was the second most urgent priority in Singapore, cited by 42% of CIOs. The technology did not rank the top three in terms of priority for IT leaders in India, where data privacy was the second most urgent priority at 46%, followed by data management and analytics at 43%.
To alleviate customers' security concerns, Lenovo applies a "zero trust" approach to the device's supply chain, embedding features that help organizations validate hardware components used in the laptop, Baiati said. These tools safeguard both software and hardware in these devices against tampering, he added.
And with 75% of data expected to be generated outside of data centers by 2025, robust connectivity on personal work devices also will be essential, according to Lenovo Singapore's general manager Nigel Lee.
5G-enabled laptops, for example, will ensure users can have seamless connection to the cloud, Lee said.
"With hybrid work firmly entrenched in Singapore, innovation that supports employee productivity and mobility is now top of mind for IT leaders," he said. No longer just about maintenance, IT products and services must facilitate this for businesses.
The Lenovo executive was speaking at the launch of the vendor's range of 5G-enabled eSIM business laptops.
News app Artifact adds AI text-to-speech voices, including Snoop Dogg and Gwyneth Paltrow Sarah Perez @sarahintampa / 7 hours
Personalized news app Artifact, developed by Instagram’s founders, is adding another new AI-powered feature today. The company is launching an AI-powered text-to-speech feature in partnership with Speechify that will allow Artifact users to listen to news articles read aloud. The feature won’t just offer a robotic-sounding voice as some other text-to-speech engines provide, but will instead introduce a variety of natural-sounding voices that can be customized by selecting different accents and audio speeds. Two celebrity voices — Snoop Dogg and Gwyneth Paltrow — are also available.
To access the new feature, Artifact users will tap the play button now at the bottom bar of any article. They can then select the voice, accent, and speed to start listening. From a slider, you can adjust the speed as slow as 0.1x to as fast as 4.5x, though most will opt for something more reasonable.
The app will allow you to continue to browse the news while the article you’re listening to continues to play in the background. The company says this will make it easier to use Artifact to catch up on the news while you’re working out, commuting, or doing chores, for example.
All of the 30+ voices are free to use and there’s no plan to charge, Artifact notes. While they’re only available in the English language for the time being, users can choose from accents like the U.K., Australia, Nigeria, and South Africa. Though Paltrow and Snoop Dogg are the only official celeb voices, there are other fun voices to choose from. For instance, one voice dubbed “Mr. President,” sounds like Obama, while “Dwight” is meant to resemble Dwight Schrute from “The Office.”
The new voices are now one of many AI-powered additions that have arrived in the news app since its February 2023 public launch.
Last month, the company announced it would also use AI to rewrite headlines of clickbait articles. When a user flags a title as clickbait, the app calls on a GPT-4 model to rewrite the headline. These rewrites are marked with a star icon to indicate they’ve been changed from the original. And in April, the app began summarizing stories using AI, even letting people pick from fun styles like “explain like I’m five,” or even just a series of emojis, for a little whimsy.
The app’s recommendation systems are also powered by AI technologies to surface content personalized to the end user, based on factors — like their clicks, reading time, dwell time, shares, and more — instead of just broadly what’s popular across the user base.
Artifact is similar in some ways to other news apps on the market, including ByteDance’s Toutiao in China, Japan-based SmartNews, and News Break, another personalized news reader with Chinese roots. But in the U.S., it also faces steep competition from other sources where users get their news including the built-in news apps on smartphones provided by Apple and Google, and even social media sources, like TikTok.
With the addition of text-to-speech, Artifact will challenge other read-it-later apps like Pocket, Matter, and Instapaper, too.
Artifact is well-liked by those who have tried it, sporting a 4.7-star review across 4.3K ratings on the U.S. App Store, but it’s not yet blowing up to mainstream adoption, trailing the charts at No. 124 in the News category.
The new AI voices will roll out in an app update starting today.
The tech behind Artifact, the newly launched news aggregator from Instagram’s co-founders
A stage agnostic VC firm, Andreessen Horowitz aka a16z, managed to capitalise on the blockchain, crypto and metaverse bubble at the right time, a formula they wish to replicate with AI. The firm seems to have steered away from the things of the past and is speeding towards generative AI with the overwhelming outpouring of its “vision for the AI-enabled future”, suggesting ways generative AI could help with “the creation of new medicines”, “climate change” and whatnot.
A16z says it takes a long view of relationships and defines its investment philosophy as being “in the business of investing in innovators”. It also claims that it “think(s) far more about how big the outcome will be if a deal succeeds than all the ways that it can fail.”
And that’s true for them, however, they apply a very crafty and effective tactic—hype, exit, repeat. It is well-known for leveraging effective PR to create hype around companies like Coinbase, Airbnb, Affirm, Instacart, Netscape, and Skype, leading to high valuations. However, after the VCs exit, the valuations of these companies often decline. If the barrage of their well-crafted and researched blogs around GenAI are anything to go by, Andreessen Horowitz is looking to do the same with generative AI.
While, Geoffrey Hinton, the Godfather of AI broke his decades-long association with Google to warn people openly about the dangers of AI, Marc Andreessen, one of the co-founders of the VC firm went ahead and wrote a 7000-word blog last month on “why AI will save the world” and all the naysayers should just shut up.
After extensively dismissing the worries of AI sceptics, Andreessen introduced a new panic: the possibility of American AI losing ground to China’s dystopian concept of AI dominance. He suggests that to prevent this, the ‘free world’ must invest heavily in developing AI as an open and unrestricted technology.
Sounds more like a move to coerce people into directing funding towards AI hype companies he is associated with, doesn’t it? In a recent episode of the Joe Rogan podcast, Marc doubled down on the usefulness of AI, in contrast to detractors who fear its potential to destroy or control humanity. He presents a compelling vision of AI’s ubiquity, envisioning its application in various fields, such as education, scientific research, art, and military strategy.
Cofounder Marc Andreessen suggested that AI will take over everything from tutoring children and helping scientists to augmenting artists’ work and improving warfare. He went to length envisioning the AI his son will grow up. He said that when children who grow up with AI go to college or enter the workforce, “they’ll have an ally right with them. They’ll have basically a partner whose goal in life will be to make them as happy and satisfied and successful as possible.”
Going All in on AI
Just yesterday, the firm announced the onboarding of Anjney Midha, Ubiquity6 cofounder and former CEO, who is joining the firm to lead its artificial intelligence investing efforts. The firm has raised a total of $7.1 billion across seven funds, which looks to support tech entrepreneurs and offer guidance from a diverse team of engineers, executives, industry experts, and academics. It has actively led fundraising campaigns, such as raising $30 million for People.ai.
a16z has raised a total of $32.4B across 27 funds in total, with 1,416 investments and 625 lead investments. It has made 203 exits in firms like Github, Coinbase, Pinterest Reddit, and also picked up new shares in OpenAI, Replit, and Replicate. Additionally, the company has dedicated a bio fund focused on advancing AI in the medical domain to aid disease prevention and treatment. Some of their notable investments include Shield AI, which develops AI software for intelligence, surveillance, and reconnaissance, and Freenome, an AI-driven platform for understanding the immune system. It has also invested in Inflection AI, Scale AI, Hippocratic AI, and Character.AI.
Not just that, a16z has also invested in Open AI CEO Sam Altman’s risky bet–-Worldcoin, which is out to scan every iris in the world. At this point, the firm seems to have leaped headlong into the fray.
The Sea of Investment
AI companies are experiencing significant attention and investment from venture capitalists. However, VCs tend to invest in AI founders who hail from from premier institutions like Stanford, Harvard and others, alongside having prior experience with big techs. Conversely, graduates from these universities can join AI startups, becoming highly valued assets for attracting funding. In exchange, these graduates would receive equity in the startup. This concept is akin to a service where Stanford University graduates act as a valuable resource for AI startups seeking investment.
This pattern has been observed in technology markets that hold disruptive potential — first with crypto and blockchain, then with the metaverse, and now with AI — the hype surrounding these technologies has attracted substantial funding. However, this surge of interest from VCs could potentially lead to the AI bubble burst. Prior to VCs’ involvement, tech giants like Google, Meta, and Amazon were already investing heavily in AI innovation and development. But once the AI hype gained momentum, even these established companies jumped on the AI bandwagon.
Despite the optimistic reports and claims, many generative AI use cases are yet to materialise, and enterprises struggle with security concerns. AI’s potential for disruption should be approached with measured scepticism, as there are numerous AI grifters seeking to capitalise on the hype.
Update: 27th July | 16:11: The article has been updated to show a16z’s investment and exit
The post Hype, Exit, Repeat: a16z’s Generative AI Journey Creates Déjà Vu appeared first on Analytics India Magazine.
If I hadn't told you that the right side of the image was generated by an AI, you probably wouldn't have known.
Using its Firefly AI technology, Adobe today announced a new Photoshop feature: Generative Expand will allow users to expand images beyond their original bounds, and fill in what's missing.
But…wait?! Photoshop already does that with Generative Fill, doesn't it?
Also: Adobe's customer experience offerings are getting a generative AI upgrade
In the company's press release, they say that 900 million images have been generated with Generative Fill — and the feature is still in beta. I'll bet that nearly every user has tried expanding the canvas using Generative Fill. So what's so special about Generative Expand?
Also: How to use Photoshop's Generative Fill AI tool to easily transform your boring photos
To answer that, let's make sure we're all on the same page by first discussing two powerful Photoshop fill commands.
Content-Aware Fill
Content-aware fill is a feature that's been around since about 2010. While it uses some intelligence and has definitely improved year over year in terms of quality, it's not using a generative AI engine to do its work.
Also: These 3 AI tools made my two-minute how-to video way more fun and engaging
Where it excels, and where I've used it extensively, is getting extensions of a textured surface to look pretty good. Take a look at this first image.
Filling the concrete and asphalt, as well as the green muck line in-between, into the white area is a lot of work using other Photoshop tools, like the rubber stamp tool. But by selecting the white area and choosing Content-Aware Fill, you get this:
You can see where the two images join, it's not quite perfect. But it gets the job done enough that a bit of retouching is all it takes to make the fill look good. Five minutes instead of an hour.
Content-Aware Fill, ironically enough, isn't really aware of its content. Oh, it knows that there are bit patterns it wants to reflect, but it has no idea that you're looking at concrete and asphalt.
Take this next example.
Here, we have a nice Oregon barn with a fence and a tree branch. Using Content-Aware Fill, we get this:
It doesn't know that the ground is the ground, the fence is the fence, or the sky is the sky. What it sees are green pixel textures, some white pixel textures, and some bluish pixel textures. As such, you wind up with the grass making it all the way up into the sky.
Content-Aware Fill lets you erase areas of the image where you don't want it to consider pixels, so for the above, I erased the barn. If you just select the white rectangle and don't erase the barn, Content-Aware Fill shows its complete lack of awareness for the content of the image:
Generative Fill
Generative Fill, which is still in beta for Photoshop, is in a completely different league. Here's how Generative Fill handled filling in the white area:
Notice how the ground is on the ground. The fence extends to the right of the tree and then behind the tree. The tree casts a shadow, including on the fence. There's even another fenced-in area behind the tree that looks as if it could well be part of the scene.
Yes, the tree is a little anemic, but that can be fixed with more generative fills near the branches. And yes, the main trunk of the tree has a little green in it, but that's a pretty simple retouch.
Perfect, it's not. But if I hadn't told you that the right side of the image was generated by an AI, you probably wouldn't have known.
Expanding on Generative Fill
Almost everyone who has experimented with Generative Fill has done this. Take a base image, increase the canvas size, and then select the white space. Like this:
The red dashes are there so that you can more easily see the area selected.
Then, once there's a selection, all you need to do is click the Generative Fill button. What was once white is filled in with the rest of the scene:
The results are amazing. You can see how it added trees, extended shadows, and even put a building in the background where it makes sense to have a building. All the additions are aware of the direction of the sun, and the shadow generation is spot on.
Generative Expand
The newly announced Generative Expand does exactly what I just showed you with Generative Fill, but does it using a different tool and a slightly streamlined workflow.
My method took three steps, three tools, a bunch of clicks, and some typing.
Also: The best AI art generators: DALL-E 2 and other fun alternatives to try
Select Canvas Size from the Image menu and type in the new dimensions, choosing to center the original image. This creates the white block.
Choose the Magic Wand tool and click on the white space. This selects the white space area.
Choose Modify→Expand from the Select menu and type in a pixel value. This creates some overlap with the original image (which makes Generative Fill work more reliably).
Generative Expand uses the Crop tool. Here's how Adobe describes its new functionality:
First, select the Crop tool and drag beyond the image's original canvas to your desired aspect ratio.
Generated content can be added with or without a text prompt. Without a prompt, click "Generate" in the Contextual Task Bar, and Photoshop will fill in the new white space with generated content that seamlessly blends with the existing image.
When using a prompt, the image will be expanded and will include the content you entered into the prompt. Select your favorite variation, and the expanded image will be added non-destructively in a new Generative Layer.
Essentially, we've all been doing Generative Expand, but now there's a version of the feature that works within the Crop tool and may save some steps.
Availability and languages
Adobe announced that the Generative Expand functionality will be available in the Photoshop beta being updated today.
The company also announced that you're now able to talk to the AI in 100 languages. They include Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani (Latin), Bangla, Bashkir, Basque, Bosnian (Latin), Bulgarian, Cantonese (Traditional), Catalan, Chinese (Literary), Chinese Simplified, Chinese Traditional, Croatian, Czech, Danish, Dari, Divehi, Dutch, English, Estonian, Faroese, Filipino, Finnish, French, French (Canada), Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Inuktitut, Irish, Italian, Japanese, Kannada, Kazakh, Khmer, Konkani, Korean, Kurdish (Central), Kurdish (Northern), Kyrgyz (Cyrillic), Lao, Latvian, Lithuanian, Macedonian, Maithili, Malagasy, Malay (Latin), Malayalam, Maltese, Marathi, Mongolian (Cyrillic), Myanmar (Burmese), Nepali, Norwegian, Odia, Pashto, Persian, Polish, Portuguese (Brazil), Portuguese (Portugal), Punjabi, Romanian, Russian, Serbian (Cyrillic), Serbian (Latin), Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Swahili, Swedish, Tamil, Tatar, Telugu, Thai, Tibetan, Tigrinya, Turkish, Ukrainian, Upper Sorbian, Urdu, Uyghur, Uzbek (Latin), Vietnamese, Welsh.
The best AI chatbots: ChatGPT and other noteworthy alternatives
That's how you know it's an AI. The only two people I know who speak that many languages are Hoshi Sato and Saru, and neither has been born yet (and, technically, never will be because they're fictional Star Trek characters).
More to come
Before I close, I want to remind you of two particularly valuable differences between Adobe's Generative Fill/Expand and tools like Midjourney.
First, Adobe lets you select an area of the image and tell it what you want to be filled. This article shows you exactly how and why that's so powerful. With Midjourney, you have to create the entire image from text.
Second, and probably the most important aspect of all of Adobe's generative AI offerings: the images generated are suitable for use without copyright concerns. Most AI image generators trained their AIs on all sorts of random and unspecified images on the Internet.
Also: The 7 best photo editing apps: From beginners to pros
As such, you may well be using part of a copyrighted image in your output if you use one of those tools. But Adobe only used images in its own Adobe Stock collection, all of which it owns and licenses as part of your Creative Cloud license. So you're very safe if you use a Photoshop-generated image and very exposed if you use something from one of the other AIs.
Adobe says it will be introducing more generative AI goodness as Photoshop's new version gets closer to formal release.
So there you go. Live long and generate.
You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter on Substack, and follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.
Google Health, Google Deepmind and Google AI have unveiled Med-PaLM M, a large multimodal generative model that flexibly encodes and interprets biomedical data. It can handle various types of medical data, including clinical language, medical images, and genomics, and performs well on a wide range of tasks, all using the same set of model weights.
Medicine is inherently multimodal. Thrilled to share Med-PaLM M, the first demonstration of a generalist multimodal biomedical AI system with a stellar team @GoogleAI @GoogleDeepMind @GoogleHealth Paper: https://t.co/oEMXZSW2bK pic.twitter.com/ZgEtG0gXEs
— Vivek Natarajan (@vivnat) July 27, 2023
It was built by fine tuning and aligning PaLM-E, a language model from Google AI, to the medical field using a specially curated open-source benchmark called MultiMedBench. MultiMedBench consists of 7 biomedical data types and 14 diverse tasks, such as medical question-answering, generating radiology reports, and identifying genomic variations. With over 1 million samples, this benchmark encourages the development of generalist biomedical AI systems.
Med-PaLM M excels in all tasks on MultiMedBench, often outperforming specialist models by a significant margin and even surpassing PaLM-E, proving the importance of adapting the model to biomedical data.
The key idea behind building a large-scale biomedical AI is to use language as a common framework for different tasks. This allows the AI to combine knowledge from various sources and transfer skills across tasks more effectively.
Excitingly, preliminary evidence suggests that Med-PaLM M can generalise to new medical tasks and concepts and perform multimodal reasoning without specific training. It can accurately identify and describe medical conditions in images using only language-based instructions and prompts, even if it has never seen such cases before.
To assess the practical use of Med-PaLM M in clinical settings, radiologists evaluated AI-generated reports at different scales. The AI’s error rate was found to be comparable to that of radiologists from previous studies, indicating its potential clinical usefulness. The big tech launched the first version of MedPaLM in December, 2022.
Read more: Responsible AI Takes Center Stage at Google I/O Connect
Google’s Unwavering Commitment to AI in Healthcare
Google’s Med-PaLM 2, a medical chatbot that answers medical questions, has been a fan favourite since its launch.
Med-PaLM 2 is built upon Google’s language model, PaLM 2, and uses LLMs tailored to the medical domain. The AI has demonstrated impressive performance on medical question-answering datasets, achieving high accuracy on the US Medical Licensing Examination (USMLE)-style questions and the Indian AIIMS and NEET medical examination questions.
Google acknowledges the complexity of personalised medical care and recognises that Med-PaLM 2’s results may not be generalisable to every medical question-answering setting and audience. The AI is trained on medical Q/A datasets but excludes patients’ personal data to adhere to ethical norms.
While having access to patients’ personal data could enhance Med-PaLM 2’s efficiency, privacy concerns are likely to prevent many patients from sharing such information. Google ensures that customers testing Med-PaLM 2 will retain control of their data in encrypted settings, inaccessible to the tech company, and the AI program will not ingest any of that data.
Read more: Google Takes AI Healthcare in Its ‘Med PaLM’
The post Google Unveils Multimodal Generative AI Model Med-PaLM M for Healthcare appeared first on Analytics India Magazine.
Python is the most popular language for data analytics, machine learning, and automation tasks due to its simplicity, vast library of data science tools like NumPy and Pandas, integration with Jupyter Notebooks which allows easy experimentation and visualization, and versatility for a wide range of uses, making it the ideal language for beginners to learn when first getting into data science.
If you are just starting out in your data science career, I highly recommend getting started with Python and its most popular data science libraries like NumPy, Pandas, Matplotlib, and Scikit-Learn. Learning Python along with these libraries will give you a solid foundation to get things done efficiently and without too many headaches, setting you up for success as you progress in data science.
2. SQL
Learning SQL is crucial for anyone working with data. You will use it to extract and analyze information from SQL databases, and it is a fundamental skill for data professionals. By understanding SQL, you can interact with relational database management systems such as MySQL, SQL Server, and PostgreSQL to retrieve, organize, and modify data effectively.
The basics of SQL include the ability to select specific data using the SELECT statement, insert new data with the INSERT statement, update existing data using the UPDATE statement, and delete data that is old or invalid using the DELETE statement.
3. Bash
Bash/Shell are not traditional programming languages, they are invaluable tools for working with data. Bash scripts allow you to string together commands to automate repetitive or complex data tasks that would be tedious to perform manually.
Bash scripts can be used to manipulate text files by searching, filtering and organizing data. They can automate ETL pipelines to extract data, transform it and load it into databases. Bash also allows you to perform calculations, splits, joins and other operations on data files from the command line and interact with databases using SQL queries and commands.
4. Rust
Rust is an up-and-coming language for data science thanks to its strong performance, memory safety, and concurrency features. However, Rust is still relatively new for data applications and has some disadvantages compared to Python.
Being a younger language, Rust has far fewer libraries for data science tasks than Python. The ecosystem of machine learning and data analysis libraries still needs to mature in Rust, meaning most codebases must be written from scratch.
However, Rust's strengths, like performance, memory, and thread safety, make it a good fit for building efficient and reliable backends for data science systems. Rust is well-suited for low-level code optimizations and parallelization needed in some data pipelines.
5. Julia
Julia is a programming language specifically created for scientific and high-performance numerical computing. One of its unique features is the ability to optimize code during the compilation process, which enables it to perform as well as, or even better than, C programming language. Additionally, Julia's syntax is inspired by popular programming languages like MATLAB, Python and R, making it easy for data scientists already familiar with these languages to learn.
Julia is open source and has a growing community of developers and data scientists contributing to its ongoing improvement. Overall, Julia provides a great balance of productivity, flexibility and performance — making it a valuable tool for data scientists, particularly those working on performance-constrained problems.
6. R
R is a popular programming language that is widely used for data science and statistical computing. It is well-suited for data science because it has a wide range of built-in functions and libraries for data manipulation, visualization, and analysis. These functions and libraries allow users to perform a variety of tasks, such as importing and cleaning data, exploring data sets, and building statistical models.
R is also known for its powerful graphics capabilities. The language includes a variety of tools for creating high-quality graphs and visualizations, which are essential for data exploration and communication.
7. C++
C++ is a high-performance programming language that is widely used for building high performance complex machine learning applications. Although it is not as commonly used in data science as some other languages like Python and R, C++ has several features that make it an excellent choice for certain types of data science tasks.
One of the key advantages of C++ is its speed. C++ is a compiled language, meaning that code is translated into machine code before it is executed, which can result in faster execution times than interpreted languages like Python and R.
Another advantage of C++ is its ability to handle large data sets. C++ has low-level memory management capabilities, which means that it can efficiently work with very large data sets without running into memory issues that can slow down other languages.
8. Scala
If you're looking for a programming language that is cleaner and less wordy than Java, then Scala might be a great option for you. It's a versatile and flexible language that combines object-oriented and functional programming paradigms.
One of the main benefits of Scala for data science is its ability to seamlessly integrate with big data frameworks like Apache Spark. This is because Scala runs on the same JVMs as these frameworks, making it a great choice for distributed big data projects and data pipelines.
If you're aiming for a career in data engineering or database management, learning Scala will help you excel in your career. However, as a data scientist, it is not necessary to acquire knowledge in this language.
Conclusion
In conclusion, if you are interested in data science, learning one or more of these eight programming languages can help kickstart or advance your career in this field. Each language offers its own unique set of advantages and disadvantages, depending on the specific data science task you are trying to accomplish.
When it comes to programming languages for data science, Python is a popular choice due to its user-friendly features, versatility, and strong community support. Other languages such as R and Julia are also great options, offering excellent support for statistical computing, data visualization, and machine learning. C++ and Rust are recommended for those in need of high-performance and memory management capabilities. Bash scripts are useful for automation and data pipelines. Lastly, it's important to learn SQL as it is a compulsory language for any tech job. Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.
More On This Topic
Data Science Programming Languages and When To Use Them
KDnuggets™ News 22:n06, Feb 9: Data Science Programming Languages and…
Programming Languages for Specific Data Roles
Top Programming Languages and Their Uses
KDnuggets™ News 22:n04, Jan 26: The High Paying Side Hustles for Data…
Whenever a new technology emerges, the immediate concern is job losses. Historically, while technology has washed away many jobs, it has also created various new job opportunities. In the age of generative AI, the discussions are no different, albeit, the impact could be different this time.
Many industry experts that AIM has spoken to are of the opinion that generative AI will be a productivity booster and will make an organisation’s workforce more efficient. However, in a recent interview with The Atlantic, OpenAI CEO Sam Altman, said, “A lot of people working on AI pretend that it’s only going to be good; it’s only going to be a supplement; and no one is ever going to be replaced,” he continued, “Jobs are definitely going to go away, full stop.”
Interestingly, Altman is not the only CEO of a prominent generative AI company claiming AI will lead to job losses. Not so long ago, Emad Mostaque, the CEO of Stability AI, on a call with UBS analysts, claimed that most of India’s outsourced coders will lose their jobs because of AI.
Entry-level coding jobs could go away
ChatGPT (powered by OpenAI’s GPT models) can code and how! In fact, the chatbot cleared a Level-3 coding interview at Google. Large language models (LLMs) are trained on huge amounts of human-written code and are able to generate a lot of code that previously required human intervention. Given LLM’s ability to generate them in a significantly faster way, entry-level coding jobs could be displaced.
Natarajan Radhakrishnan, president and CIO, Hinduja Global Solutions, believes that AI will not take away coding jobs entirely, however, “The lower end of coding may indeed face replacement, not just in India, but globally”, he told AIM.
With the rise of generative AI, lower-level coding tasks have diminished in value, as LLMs can easily generate cookie-cutter code, rendering such tasks unnecessary, according to Anurag Sahay, managing director – AI and Data Science, at Nagarro. “Coders and engineers will have to bring two important value propositions to the table – an ability for complex system engineering and an ability to leverage LLMs quickly to create digital assets,” he told AIM.
Moreover, in India, more than 15 lakh engineers graduate every year from over 2500 institutions. Despite the huge churn-out, reports over the years have suggested that a huge chunk of them are unfit for hiring. A McKinsey report pegged that number at a massive 94%. In the current AI landscape, this situation can be concerning. Companies may prioritise AI assistance to accomplish tasks. And while they may not lay off inexperienced coders, they might reconsider hiring them in the future.
Gibin Varghese, partner-technology practice, at WalkWater Talent Advisors, concurs. According to him, generative AI will definitely reduce the demand for entry-level coding jobs. “In the future, the demand for coders with knowledge or experience of working with generative AI tools will increase. Developers who haven’t adapted to generative AI tools will be at risk of losing their jobs. Freshers will need to upskill in generative AI to remain relevant,” he told AIM.
Entry-level coding quality will be transformed
However, not all is gloomy for entry-level coding job seekers, be it in India, or any part of the world. As Varghese mentioned, only developers who fail to adapt to newer technologies are at risk. “Hence entry-level developers can upskill in emerging areas to stay relevant and thrive in the evolving job market,” Sekhar Garisa, CEO, Foundit (previously Monster APAC & ME), told AIM.
In fact, the very nature of entry-level coding jobs is going to change in the days to come. “A lot of HTML, Web, and Mobile developers developing template code will have to develop meta and thinking skills along with more software engineering skills to remain relevant. So, we expect better-quality entry-level coders. Cookie-cutter entry-level coding skills will not stay relevant,” Sahay said.
Currently, numerous online resources and generative AI courses empower individuals to upskill and enhance their knowledge. Educational institutions will likely introduce similar courses to develop the skills of countless students enrolling each year. “We expect educational institutes to pick up these tools and teach students skills to use these AI tools and add significant value to the AI creatives. This is the only future that looks promising,” Sahay added.
World will write way more software than ever before
While entry-level coding jobs might face challenges, the demand for skilled coders will remain because of the necessity for building functional, high-quality software. “The world is becoming more digital and there is a need for more software than ever before. It is our belief that the world will be writing way more software than ever and the demand for quality software engineers who know how to use these AI tools to build complex software outcomes will only get higher,” Sahay said.
With generative AI coming into the picture, the overall expectation around the productivity of a coder is also changing drastically. Today, a website can be built within an hour, compared to a month’s time earlier. The same is true of test cases that needed to be written or documentation around code. These creatives have been made vastly simpler with LLMs.
However, in the long run, as these LLMs further mature, there could be a scenario where these models can perform complex coding tasks and maybe this is the scenario Altman envisages. The developments happening in this space are fast and it’s difficult to predict. Nonetheless, even if it does, technology does create jobs, and hence, so will generative AI.
The post Coders Can be Gen AI Adopters or Job Losers appeared first on Analytics India Magazine.
The success of machine learning (ML) across many domains has brought with it a new set of challenges – specifically the need to continuously train and evaluate models and continuously check for drift in training data. Continuous integration and deployment (CI/CD) is at the core of any successful software engineering project and is often referred as DevOps. DevOps helps streamline code evolution, enables various testing frameworks, and provides flexibility for enabling selective deployment to various deployment servers (dev, staging, prod, etc.).
The new challenges associated with ML have expanded the traditional scope of CI/CD to also include what is now commonly referred as Continuous Training (CT), a term first introduced by Google. Continuous training requires ML models to be continuously trained on new datasets and evaluated for expectations before being deployed to production, as well as enabling many more ML specific features. Today, within a machine learning context, DevOps is becoming known as MLOps and includes CI, CT & CD.
MLOps Principles
All product development is based on certain principles and MLOps is no different. Here are the three most important MLOps principles.
Continuous X: The focus of MLOps should be in evolution, whether it is continuous training, continuous development, continuous integration or anything that is continuously evolving/changing.
Track Everything: Given the exploratory nature of ML, one needs to track and collect whatever happens, similar to the processes in a science experiment.
Jigsaw Approach: Any MLOps framework should support pluggable components. However, it’s important to strike the right balance: too much pluggability causes compatibility issues, whereas too little restricts the usage.
With these principles in mind, let’s identify the key requirements that govern a good MLOps framework.
MLOps Requirements
As previously mentioned, Machine learning has driven a new unique set of requirements for Ops.
Reproducibility: Enable ML experiments to reproduce the same results repeatedly to validate the performance.
Versioning: Maintain versioning from all directions, including: data, code, models and configs. One way to perform ‘data-model-code’ versioning is to using version control tools like GitHub.
Pipelining: Although Directed Acyclic Graph (DAG) based pipelines are often used in non-ML scenarios (ex -Airflow), ML brings its own pipelining requirements to enable continuous training. Reusability of pipeline components for train and predict ensures consistency in feature extraction and reduces data processing errors.
Orchestration & Deployment: ML model training requires a distributed framework of machines involving GPUs and therefore, executing a pipeline in the cloud is an inherent part of the ML training cycle. Model deployment based on various conditions (metric, environment etc.) brings unique challenges in machine learning.
Flexibility: Enable flexibility for choosing data sources, selecting a cloud provider and deciding upon different tools (data analysis, monitoring, ML frameworks, etc.) Flexibility can be achieved by providing an option for plugins to external tools and/or offering the capability to define custom components. A flexible orchestration & deployment component ensures cloud agnostic pipeline execution and ML service.
Experiment Tracking: Unique to ML, experimentation is an implicit part of any project. After multiple rounds of experimentation (i.e. experimentation with architecture or hyper-parameters in the architecture), an ML model gets matured. Keeping a log of each experiment for future reference is essential to ML. Experiment tracking tools can be used to ensure code and model versioning and DVC like tools ensure code-data versioning.
Practical Considerations
In the excitement of creating ML models, some specific ML hygiene is often missed: such as initial data analysis or hyperparameter tuning or pre-/post- processing. In many cases, there is a lack of an ML production mindset from the beginning of the project, which leads to surprises (memory issues, budget overflow etc.) at later stages of the project, especially during production time, resulting in re-modeling and delayed time-to-market. But using an MLOps framework from the beginning of a ML project addresses production considerations early on and enforces a systematic approach to solving machine learning problems such as data analysis, experiment tracking etc.
An MLOps also makes it possible to be production-ready at any given point of time. This is often crucial for startups when there is a requirement for shorter time-to-market. With MLOps providing flexibility in terms of orchestration & deployment, production readiness can be achieved by pre-defining orchestrators (ex- github action) or deployers (ex- MLflow, KServe etc.) which are part of MLOps pipelining.
Existing Frameworks for MLOps
Cloud service providers like Google, Amazon, Azure provide their own MLOps frameworks that can be used in their own platform or as part of existing machine learning frameworks (TFX pipelining as part of Tensorflow framework). These MLOps frameworks are easy to use and exhaustive in their functionality.
Using an MLOps framework from a cloud service provider restricts an organization to use MLOps in their environment. For many organizations this becomes a big restriction as usage of cloud service depends on what their customer wants. In many cases, one needs an MLOps framework that provides flexibility in terms of choosing a cloud provider and at the same time, has most of the functionalities of MLOps.
Open-source MLOps frameworks come in handy for such scenarios. ZenML, MLRun, Kedro, Metaflow are some of well-known open-source MLOps frameworks widely used with their own pros and cons. They all provide good flexibility in terms of choosing cloud providers, orchestration/deployment and ML tools as part of their pipeline. Selecting of any of these open source frameworks depends on the specific MLOps requirements. However, all these frameworks are generic enough to cater to wide range of requirements.
Based on experience with these open-source MLOps frameworks in their current state, I recommend the following:
Adopt MLOps Early On
MLOps is the next evolution in DevOps and is bringing together people from different domains: data engineers, machine learning engineers, infrastructure engineers as well as others. In the future we can expect MLOps to become low-code, similar to what we’ve seen within DevOps today. Startups in particular should adopt MLOps in their early stages of development to ensure faster time-to-market in addition to the other benefits it brings to the table. Abhishek Gupta is the Principal Data Scientist at Talentica Software. In his current role, he works closely with a number of companies to help them with AI/ML for their product lineups. Abhishek is an IISc Bangalore alumnus who has been working in the area of AI/ML and big data for more than 7 years. He has a number of patents and papers in various areas like communication networks and machine learning.
More On This Topic
Production-Ready Machine Learning NLP API with FastAPI and spaCy
Managing Model Drift in Production with MLOps
3 mindset changes to become a better analyst
Data-Centric AI: Is it Real? For Everyone? Are We Ready?
On-device AI with Developer-Ready Software Stacks
Ready for #2022Analytics in Houston? The Networking, Connections, and…
Adopt MLOps Early On