Artists across industries are strategizing together around AI concerns

Artists across industries are strategizing together around AI concerns Taylor Hatmaker @tayhatmaker / 3 days

As creative industries grapple with AI’s explosion into every artistic medium at once, separate calls from artists warning the world to take action before it’s too late are starting to converge. From fake Drake songs to stylized Instagram profile pictures, art conjured with newly sophisticated AI tools is suddenly ubiquitous — and so are conversations about how to rein in the technology before it does irrevocable harm to creative communities.

This week, digital rights organization Fight for the Future partnered with music industry labor group United Musicians and Allied Workers to launch #AIdayofaction, a campaign that calls on Congress to block corporations from obtaining copyrights on music and other art made with AI.

The idea is that by preventing industry behemoths like major record labels, for example, from copyrighting music made with the assistance of AI, those companies will be forced to keep looping humans into the creative process. But those same concerns — and the same potential strategies for pushing back against the onslaught of AI — exist across creative industries.

“It’s funny because if you’ve talked to musicians who have these concerns, they say, ‘Well, authors have been very quiet.’ If you talk to others about these concerns, they say, ‘Well, musicians and photographers don’t seem to care at all,’” Fight for the Future Campaigns and Communications director Lia Holland told TechCrunch. “So part of it also is that the different creative fields, when it comes to this sort of work, are a little bit siloed.”

“That was another intent with our launching this effort with the day of action, to try to illustrate how these are common concerns that are shared across artistic mediums. And to create an organizing point . . . because when artists of different mediums move together they have a lot more power.”

The campaign targets potential corporate abuse of AI technology, but it’s realistic about the ways that musicians and some other creatives could benefit on an individual level from automating parts of their work. The goal is that AI tools “become ways for individual humans to make more money, work less, and compete with the corporations that exploit them.”

“It’s really interesting from a music perspective, specifically, because . . . musicians are perhaps more familiar with the idea of AI,” Holland said. “Musicians in general are more familiar with things like music production software, and AI tools like MIDI drum loops . . . so I think that there is a certain amount of more progressive learning from them, when it comes to technology, and its ability to make their music better.”

When it comes to art and AI, the conversation is complicated, to say the least. Musicians are nervous about industry giants copyrighting AI music and cutting them out of the process. Major record labels are worried about AI models training on their catalogues and stealing a slice of their considerable pie. Spotify erased thousands of AI-crafted songs from its platform but also recently globally launched an AI-powered DJ that curates music for listeners while talking to them in a synthetic voice.

A new Drake x The Weeknd track just blew up — but it’s an AI fake

“The training of generative AI using our artists’ music . . . begs the question as to which side of history all stakeholders in the music ecosystem want to be on: the side of artists, fans and human creative expression, or on the side of deep fakes, fraud and denying artists their due compensation,” Universal Music Group said after a song using AI to imitate Drake and The Weeknd, two of its artists, went viral.

These same conversations and contradictions are manifesting across creative industries, but artists themselves don’t always have a seat at the table. Independent artists in particular are learning that their voices resonate louder when coming together across disciplines to push back against what Holland describes as an “extraordinary spectrum of exploitation” that leverages their work.

In a roundtable hosted by the FTC this week, the agency brought together figures from across creative industries — from voice acting and science fiction to screenwriting, music, illustration and even fashion — to delve into how generative AI is affecting creatives.

“I know that generative AI in particular poses a unique set of opportunities and challenges to creative industries,” FTC chair Lina Khan said. “We’ve already heard significant concerns about how these technologies could virtually overnight significantly disempower creators and artists who may watch their life’s creation be appropriated into models over which they have no control.”

In the comments, representatives from myriad creative communities expressed concerns around opt-out requirements that by default train AI models on artists’ original work, and how existing copyright law could be a useful if not comprehensive tool for setting out regulatory guardrails.

In the conversation, a representative with the WGA emphasized that while striking writers obtained their own protections in a newly won agreement, the fight for artists’ livelihoods “doesn’t stop at the bargaining table.”

Whether Congress mobilizes in time to address mounting concerns around AI and creative industries or not, for its part the FTC does appear to be very tuned into the technology’s risks — and the power of bringing voices together across industries.

“Art is fundamentally human,” FTC commissioner Rebecca Slaughter said.

“Humans may use technology to assist in creating art, but something cannot be art without human input. Technology is, by definition, not human . . . humans may endeavor to make generative AI that is ever more intelligent, [but] it cannot and will not replace human creativity.”

The writers’ strike is over; here’s how AI negotiations shook out

How much can artists make from generative AI? Vendors won’t say

Open source isn’t ready for generative AI. How stakeholders are changing this light bulb together

bulbs-gettyimages-486549765

Without open source, there is no AI. It's that simple. But, those same licenses have been showing their age: The Gnu General Public License (GPL), Apache License, and Mozilla Public License don't fit well with software-as-a-service or cloud services. AI poses even larger problems. The open-source licenses, with their copyright law foundations, aren't a good fit for AI's large language models (LLM)s.

This isn't just some theoretical techno-legal issue, either. It's already showing up in the courts.

Also: Open source is actually the cradle of artificial intelligence. Here's why

In J. Doe 1 et al. vs GitHub, the plaintiffs allege that Microsoft, OpenAI, and GitHub — via their commercial AI-based system, OpenAI's Codex and GitHub's Copilot — stole their open source code. The class action suit claims that code "suggested" by AI often consists of near-identical strings of code scraped from public GitHub repositories — but without the required open-source license attributions.

On a related issue, two groups of writers, including George R.R. Martin, Michael Chabon, and John Grisham, are suing Microsoft and OpenAI for taking their work and using it in their LLMs. Copyright, the legal foundation of open source, is at the heart of this issue.

But this isn't simply a Microsoft problem.

As Sean O'Brien, Yale Law School lecturer in cybersecurity and founder of the Yale Privacy Lab, told my ZDNET colleague David Gewirtz: "I believe there will soon be an entire sub-industry of trolling that mirrors patent trolls, but this time surrounding AI-generated works. A feedback loop is created as more authors use AI-powered tools to ship code under proprietary licenses. Software ecosystems will be polluted with proprietary code that will be the subject of cease-and-desist claims by enterprising firms."

Others, like German researcher and politician Felix Reda, claim that all AI-produced code is public domain.

US attorney Richard Santalesa, a founding member of the SmartEdgeLaw Group, told Gewirtz that there exist both contract and copyright law issues — and they're not the same thing. Santalesa believes companies producing AI-generated code will "as with all of their other IP, deem their provided materials – including AI-generated code – as their property." Besides, public domain code is not the same thing as open-source code.

Also: Red Hat's new rule: Open source betrayal?

So, what's to be done? Simply claiming your AI is open source is a nonstarter. Meta, for example, claims Llama 2 is open source. It's not.

As Erica Brescia, a managing director at RedPoint, the open source-friendly venture capital firm, asked on Twitter: "Can someone please explain to me how Meta and Microsoft can justify calling Llama 2 open source if it doesn't actually use an OSI [Open Source Initiative]-approved license or comply with the OSD [Open Source Definition]? Are they intentionally challenging the definition of OSS [Open Source Software]?"

Here's the short explanation: Meta is using open source as a marketing term, not a legal one. That usage won't fly once the lawsuits mount up

The problem with Llama 2 specifically is that it blocks extremely profitable companies from using it. According to Stephen O'Grady, open-source licensing expert and RedMonk co-founder, the problem is that they won't work in open source. "Imagine if Linux was open source unless you worked at Facebook,"

Also: Red Hat's new rule: Open source betrayal?

At the same time, as OpenUK CEO Amanda Brock observed, "I don't think we're going to see going forward any LLM or any significant AI being able to be licensed as open source, because the key to open source is the Open Source Definition."

And the road to that Definition was a long and bumpy one.

The first free software licenses began In the early 1980s when MIT Lab programmer Richard M. Stallman couldn't get an early laser printer, the Xerox 9700, to produce error messages. The problem? Stallman couldn't read or change its source code. At the time, this was a new development. Although we now think of proprietary software as the default, it wasn't then.

So, Stallman created the GNU General Public License (GPL). While not the first Free Software license (that honor belongs to the Berkeley Software Distribution (BSD) license), GNU would prove to be very influential. In no small part, that's because Linus Torvalds chose to use the GPLv2 as Linux's license.

The GPL is based on two principles. First, software code can be copyrighted. Second, anyone is free to read and edit the code so long as these freedoms aren't taken away from anyone else.

Also: A look back at 40 Years of GNU and the Free Software Foundation

By 1985, Free Software was becoming popular, but it also had become clear that the word "free" was too ambiguous. After Netscape released Mozilla's source code — which became the basis of the Firefox web browser — several leading Free Software luminaries, including Eric S. Raymond, Bruce Perens, Michael Tiemann, Jon "Maddog" Hall, and Christine Peterson, coined the phrase open source to describe this kind of license. In 1998, Perens and Raymond went on to found the OSI, which drafted the Open Source Definition (OSD) and used this as the general guide to defining all open-source licenses.

All open-source licenses must comply with the OSD. For AI and LLMs, that's much easier said than done.

True, there are open LLMs such as Falcon, FastChat-T5, and OpenLLaMA. But most LLMs contain proprietary, copyrighted, or simply unknown information that their owners won't tell you about. The Electronic Frontier Foundation (EFF) says it well: "Garbage In, Gospel Out."

We've seen this problem coming for a while. At Open Source Europe in Bilbao, Spain, last month, I spoke with Stefano Maffulli, executive director of the Open Source Initiative (OSI), the organization that defines and manages open-source licenses. "The process started two years ago when GitHub Copilot came out," Maffulli told me. "It was a watershed moment. All of a sudden, code you wrote as a human for humans, everything we have produced and put on the Internet was being harvested for machine learning."

Also: The best AI chatbots: ChatGPT and alternatives

So, what can we do? Maffulli and other open-source and AI leaders are working on combining AI with open-source licenses in sensible ways.

Maffulli observed that combining AI with open-source licenses is as hard, if not harder, than when software copyright was first applied to source code in the 1980s (when Free Software and open-source were first defined). True, open-source AI programs — such as TensorFlow, PyTorch, and Hugging Face — work well with old-style licenses. But old-style software isn't the problem. It's where software and data mix that the existing open-source licenses begin to break down. Specifically, it's where all that data and code merge together in AI/ML artifacts — such as datasets, models, and weights — that's where trouble emerges. "Therefore," said Mafulli, "we need to make a new definition for open-source AI."

This must be a definition that all stakeholders can agree upon and work with. Free software and open source are no longer just matters for developers. The goals of open-source savvy programmers and lawyers aren't the same as those of AI companies. To address this, Maffulli, together with Google, Microsoft, GitHub, Open Forum Europe, Creative Commons, Wikimedia Foundation, Hugging Face, GitHub, the Linux Foundation, ACLU Mozilla, and the Internet Archive, are working on a draft for defining a common understanding of open-source AI. In other words, all the AI players are working on the definition.

If all goes well, we can expect to see the fruits of their labor as early as this month. And while this will only be the first draft of the AI Open Source Definition, I expect that it will be finalized as quickly as possible. Everyone involved knows that AI is advancing rapidly and the sooner we get an open-source framework around it, the better.

Artificial Intelligence

ChatGPT’s mobile app hit record $4.58M in revenue last month, but growth is slowing

ChatGPT’s mobile app hit record $4.58M in revenue last month, but growth is slowing Sarah Perez @sarahintampa / 8 hours

There’s some good news and bad news for ChatGPT’s mobile efforts. On the positive side, the AI chatbot continues to see the number of app installs and its revenue grow, with September setting a new record on both fronts: 15.6 million downloads and nearly $4.6 million in gross revenue across its iOS and Android apps worldwide. However, according to new data from market intelligence firm Appfigures, revenue growth has now begun to slow. While over the past couple of months, revenue growth was topping 30% — 31% in July and 39% in August — that dropped to 20% growth as of September.

Slowing revenue growth could be the first indication that ChatGPT is nearing saturation in terms of how many mobile users are willing to pay for the upgraded ChatGPT+ subscription service — a fairly hefty $19.99 per month in-app purchase that offers faster response times, priority access at peak times and early access to new features and improvements. The subscription has sold well so far, as ChatGPT on mobile brought in gross revenues of $2.1 million in June, increasing to $2.74 million in July, then $3.81 million in August, and now, the latest record, $4.58 million in September.

Surprisingly, however, ChatGPT is not the largest AI app by revenue. A competitor called Ask AI is making more thanks to heavy ad spending, rising from $6.48 million in May, when ChatGPT mobile launched, to a peak of $6.55 million in August, Appfigures’ data indicates. It dropped slightly in September, down to $5.51 million — but that’s still larger than ChatGPT. Other competitors like Genie and AI Chat Smith have not grown as big as Ask AI.

Net revenue may be a different story, given Ask AI’s ad spend, of course. As for ChatGPT, it netted around $3.2 million in September after Apple and Google took their cut of the in-app purchase revenues.

In addition to its revenue record, ChatGPT saw 15.6 million installs in September, bringing its lifetime total to 52.2 million, Appfigures estimates. Google Play is driving the downloads, accounting for 9 million in September while the App Store accounted for the other 6.6 million.

But the App Store, not surprisingly, is driving revenue — or $3 million of the total in-app purchases last month. The U.S. is the largest market for the AI chatbot app, accounting for 60% of ChatGPT’s revenues.

Multimodal AI Evolves as ChatGPT Gains Sight with GPT-4V(ision)

chatgpt vision

In the ongoing effort to make AI more like humans, OpenAI's GPT models have continually pushed the boundaries. GPT-4 is now able to accept prompts of both text and images.

Multimodality in generative AI denotes a model's capability to produce varied outputs like text, images, or audio based on the input. These models, trained on specific data, learn underlying patterns to generate similar new data, enriching AI applications.

Recent Strides in Multimodal AI

A recent notable leap in this field is seen with the integration of DALL-E 3 into ChatGPT, a significant upgrade in OpenAI's text-to-image technology. This blend allows for a smoother interaction where ChatGPT aids in crafting precise prompts for DALL-E 3, turning user ideas into vivid AI-generated art. So, while users can directly interact with DALL-E 3, having ChatGPT in the mix makes the process of creating AI art much more user-friendly.

Check out more on DALL-E 3 and its integration with ChatGPT here. This collaboration not only showcases the advancement in multimodal AI but also makes AI art creation a breeze for users.

openai.com dall-e-3

https://openai.com/dall-e-3

Google's health on the other hand introduced Med-PaLM M in June this year. It is a multimodal generative model adept at encoding and interpreting diverse biomedical data. This was achieved by fine-tuning PaLM-E, a language model, to cater to medical domains utilizing an open-source benchmark, MultiMedBench. This benchmark, consists of over 1 million samples across 7 biomedical data types and 14 tasks like medical question-answering and radiology report generation.

Various industries are adopting innovative multimodal AI tools to fuel business expansion, streamline operations, and elevate customer engagement. Progress in voice, video, and text AI capabilities is propelling multimodal AI's growth.

Enterprises seek multimodal AI applications capable of overhauling business models and processes, opening growth avenues across the generative AI ecosystem, from data tools to emerging AI applications.

Post GPT-4's launch in March, some users observed a decline in its response quality over time, a concern echoed by notable developers and on OpenAI’s forums. Initially dismissed by an OpenAI, a later study confirmed the issue. It revealed a drop in GPT-4’s accuracy from 97.6% to 2.4% between March and June, indicating a decline in answer quality with subsequent model updates.

chatgpt-ai

ChatGPT (Blue) & Artificial intelligence (Red) Google Search Trend

The hype around Open AI's ChatGPT is back now. It now comes with a vision feature GPT-4V, allowing users to have GPT-4 analyze images given by them. This is the newest feature that's been opened up to users.

Adding image analysis to large language models (LLMs) like GPT-4 is seen by some as a big step forward in AI research and development. This kind of multimodal LLM opens up new possibilities, taking language models beyond text to offer new interfaces and solve new kinds of tasks, creating fresh experiences for users.

The training of GPT-4V was finished in 2022, with early access rolled out in March 2023. The visual feature in GPT-4V is powered by GPT-4 tech. The training process remained the same. Initially, the model was trained to predict the next word in a text using a massive dataset of both text and images from various sources including the internet.

Later, it was fine-tuned with more data, employing a method named reinforcement learning from human feedback (RLHF), to generate outputs that humans preferred.

GPT-4 Vision Mechanics

GPT-4's remarkable vision language capabilities, although impressive, have underlying methods that remains on the surface.

To explore this hypothesis, a new vision-language model, MiniGPT-4 was introduced, utilizing an advanced LLM named Vicuna. This model uses a vision encoder with pre-trained components for visual perception, aligning encoded visual features with the Vicuna language model through a single projection layer. The architecture of MiniGPT-4 is simple yet effective, with a focus on aligning visual and language features to improve visual conversation capabilities.

MiniGPT-4

MiniGPT-4's architecture includes a vision encoder with pre-trained ViT and Q-Former, a single linear projection layer, and an advanced Vicuna large language model.

The trend of autoregressive language models in vision-language tasks has also grown, capitalizing on cross-modal transfer to share knowledge between language and multimodal domains.

MiniGPT-4 bridge the visual and language domains by aligning visual information from a pre-trained vision encoder with an advanced LLM. The model utilizes Vicuna as the language decoder and follows a two-stage training approach. Initially, it's trained on a large dataset of image-text pairs to grasp vision-language knowledge, followed by fine-tuning on a smaller, high-quality dataset to enhance generation reliability and usability.

To improve the naturalness and usability of generated language in MiniGPT-4, researchers developed a two-stage alignment process, addressing the lack of adequate vision-language alignment datasets. They curated a specialized dataset for this purpose.

Initially, the model generated detailed descriptions of input images, enhancing the detail by using a conversational prompt aligned with Vicuna language model's format. This stage aimed at generating more comprehensive image descriptions.

Initial Image Description Prompt:

###Human: <Img><ImageFeature></Img>Describe this image in detail. Give as many details as possible. Say everything you see. ###Assistant:

For data post-processing, any inconsistencies or errors in the generated descriptions were corrected using ChatGPT, followed by manual verification to ensure high quality.

Second-Stage Fine-tuning Prompt:

###Human: <Img><ImageFeature></Img><Instruction>###Assistant:

This exploration opens a window into understanding the mechanics of multimodal generative AI like GPT-4, shedding light on how vision and language modalities can be effectively integrated to generate coherent and contextually rich outputs.

Exploring GPT-4 Vision

Determining Image Origins with ChatGPT

GPT-4 Vision enhances ChatGPT's ability to analyze images and pinpoint their geographical origins. This feature transitions user interactions from just text to a mix of text and visuals, becoming a handy tool for those curious about different places through image data.

Chatgpt-vision-GPT-4

Asking ChatGPT where a Landmark Image is taken

Complex Math Concepts

GPT-4 Vision excels in delving into complex mathematical ideas by analyzing graphical or handwritten expressions. This feature acts as a useful tool for individuals looking to solve intricate mathematical problems, marking GPT-4 Vision a notable aid in educational and academic fields.

Chatgpt-vision-GPT-4

Asking ChatGPT to understand a complex math concept

Converting Handwritten Input to LaTeX Codes

One of GPT-4V's remarkable abilities is its capability to translate handwritten inputs into LaTeX codes. This feature is a boon for researchers, academics, and students who often need to convert handwritten mathematical expressions or other technical information into a digital format. The transformation from handwritten to LaTeX expands the horizon of document digitization and simplifies the technical writing process.

GPT-4V's ability to convert handwritten input into LaTeX codes

GPT-4V's ability to convert handwritten input into LaTeX codes

Extracting Table Details

GPT-4V showcases skill in extracting details from tables and addressing related inquiries, a vital asset in data analysis. Users can utilize GPT-4V to sift through tables, gather key insights, and resolve data-driven questions, making it a robust tool for data analysts and other professionals.

GPT-4V deciphering table details and responding to related queries

GPT-4V deciphering table details and responding to related queries

Comprehending Visual Pointing

The unique ability of GPT-4V to comprehend visual pointing adds a new dimension to user interaction. By understanding visual cues, GPT-4V can respond to queries with a higher contextual understanding.

GPT-4V-demonstrates-the-unique-capability-of-understanding-visual-pointing

GPT-4V showcases the distinct ability to comprehend visual pointing

Building Simple Mock-Up Websites using a drawing

Motivated by this tweet, I attempted to create a mock-up for the unite.ai website.

While the outcome didn't quite match my initial vision, here's the result I achieved.

ChatGPT Vision based output HTML Frontend

ChatGPT Vision based output HTML Frontend

Limitations & Flaws of GPT-4V(ision)

To analyze GPT-4V, Open AI team carried qualitative and quantitative assessments. Qualitative ones included internal tests and external expert reviews, while quantitative ones measured model refusals and accuracy in various scenarios such as identifying harmful content, demographic recognition, privacy concerns, geolocation, cybersecurity, and multimodal jailbreaks.

Still the model is not perfect.

The paper highlights limitations of GPT-4V, like incorrect inferences and missing text or characters in images. It may hallucinate or invent facts. Particularly, it's not suited for identifying dangerous substances in images, often misidentifying them.

In medical imaging, GPT-4V can provide inconsistent responses and lacks awareness of standard practices, leading to potential misdiagnoses.

Unreliable performance for medical purposes.

Unreliable performance for medical purposes (Source)

It also fails to grasp the nuances of certain hate symbols and may generate inappropriate content based on the visual inputs. OpenAI advises against using GPT-4V for critical interpretations, especially in medical or sensitive contexts.

Wrapping Up

Created using Fast Stable Diffusion XL

Created using Fast Stable Diffusion XL https://huggingface.co/spaces/google/sdxl

The arrival of GPT-4 Vision (GPT-4V) brings along a bunch of cool possibilities and new hurdles to jump over. Before rolling it out, a lot of effort has gone into making sure risks, especially when it comes to pictures of people, are well looked into and reduced. It's impressive to see how GPT-4V has stepped up, showing a lot of promise in tricky areas like medicine and science.

Now, there are some big questions on the table. For instance, should these models be able to identify famous folks from photos? Should they guess a person's gender, race, or feelings from a picture? And, should there be special tweaks to help visually impaired individuals? These questions open up a can of worms about privacy, fairness, and how AI should fit into our lives, which is something everyone should have a say in.

7 High Paying Side Hustles for Data Scientists

7 High Paying Side Hustles for Data Scientists
Image by Author

The economic landscape has changed dramatically over the past couple of years. With high inflation and fears of a recession, personal finances are tighter for many professionals. If you work in data science or a related field, you may be looking for ways to supplement your income during these trying times. The good news is that the demand for data skills remains high, and there are plentiful opportunities to earn extra money on the side.

In this article, we’ll explore seven lucrative side hustles that are well-suited for data scientists and other technical professionals. Whether you want to offer your data expertise as a consultant, create online courses, do freelance analytics work, or explore other entrepreneurial ideas, you can take advantage of the flexibility of side gigs and passive income.

1. Teaching

Teaching is still a high paying job in the data science field. You can create an online course and get paid through platforms like YouTube, DataCamp, Udemy, Skillshare, LinkedIn Learning, and 365 Data Science. Or you can become a guest lecturer at a university to teach a data science course. For those interested in continuous teaching, you can start your own academy or bootcamp to train aspiring data professionals.

7 High Paying Side Hustles for Data Scientists
Image from Teaching on DataCamp

By sharing your knowledge through structured lessons and programs, either online or in-person, you can earn a great income through teaching gigs as a side hustle. With data skills still in high demand, students will be eager to learn from experienced professionals. Teaching enables you to set your own schedule and reach a global audience of learners hungry for real-world data science expertise.

2. Freelancing

Freelancing is a great way for data science professionals to gain varied experience and get paid for their expertise. As a freelancer, you are your own boss and set the timeline and terms for projects. This control and flexibility can be highly rewarding. There are many platforms to find freelance gigs including Upwork, Fiverr, Toptal, and LinkedIn.

7 High Paying Side Hustles for Data Scientists
Image from Toptal

You can showcase your projects and capabilities on these sites and your professional network to attract clients. As you build your personal brand and portfolio, people will start reaching out with freelance opportunities. Freelancing enables you to expand your skills and work with different companies and industries.

3. Technical Writing

Technical writing on platforms like Medium and Substack can be great for your additional income. You can choose a specific niche within data science to write about regularly. As you build an audience and followers, you can monetize through payments per article view as well as subscriber recurring revenue. Many technical writers are earning six-figure incomes simply from content creation on these sites. You can also look for contract technical writing jobs from companies in need of documentation, tutorials, blogs, and other materials.

7 High Paying Side Hustles for Data Scientists
Image from Medium

The demand for quality tech writing is strong, and your insider knowledge of data science can give you an advantage. Write consistently in your chosen sub-field and promote your articles to gain paying subscribers over time. With the right niche and regular publishing, technical writing can become a steady side income stream.

4. Consultancy

You can consult for local companies to help improve their analytics capabilities, tooling, data infrastructure, and modeling. Or provide remote consulting sessions to firms across the country and world.

As a consultant, you leverage your experience to review current business challenges and provide strategic recommendations to enhance data practices. You can offer short-term consulting gigs to diagnose issues and provide a roadmap for improvement. Or serve as an ongoing advisor across longer projects to oversee implementation.

7 High Paying Side Hustles for Data Scientists
Image from Guidepoint

The demand for expert data science guidance makes consulting extremely profitable, with hourly rates often exceeding standard salaries. You can consult across many industry verticals by promoting your specialized abilities in certain domains. Over time, consulting gives you exposure to diverse business problems while allowing you to expand your expertise.

5. Participating in Competition

While it is not a steady stream of income, participating in data science competitions can be a lucrative side hustle. Platforms like Kaggle have competitions with prize pools from $60,000 to $500,000 USD and more. Even winning or placing high a couple times per year can equal or exceed your regular yearly salary.

7 High Paying Side Hustles for Data Scientists
Image from Kaggle

Though success is not guaranteed, with the right skills you can win thousands for solving real-world machine learning challenges. The key is understanding the platforms and continuously participating in competitions aligned with your expertise. While competition winnings should not replace a stable income, for those confident in their data science abilities, competitions provide opportunities to profit from your talent.

6. Career Counseling

With at least 5 years of industry experience, data professionals can offer career counseling services as a high paying side gig. You can get paid per hour to provide guidance to recent graduates or those looking to transition into data roles. Platforms like Skilled allow you to charge for mock interviews, resume reviews, and general career advice.

7 High Paying Side Hustles for Data Scientists
Image from Skilled

You can also find private clients interested in personalized coaching to break into the field. During paid sessions, you will provide insider tips on the data science recruitment and interview process. You'll recommend courses, projects, and other steps to become a strong candidate. Your real-world understanding of hiring practices and qualifications is extremely valuable. Counseling is also rewarding as you directly enable people to advance their careers.

7. Collaborating

Collaborating on projects can lead to high paying opportunities for data professionals. You may receive invites to partner on commercial projects, startups, or open source initiatives. While some collaborations are unpaid, many come with compensation, especially if you treat it like contract work. When brought onto a team for your expertise, negotiate fair pay for your time and contributions, even if you believe in the project. There are also platforms like Omdena that offer paid collaborative data science projects you can participate in.

7 High Paying Side Hustles for Data Scientists
Image from Github

Contributing to open source projects on GitHub can also open up lucrative opportunities for data professionals. Many popular open source data tools have corporate sponsors who hire from the contributor community. By consistently contributing quality code and documentation, you can get noticed and brought on board. Your open source work serves as proof of your abilities.

Outside of potential employment, active GitHub contributors can be sponsored directly through GitHub Sponsors. Popular repositories can attract financial backing from users who rely on the project. Even without direct sponsorship, your open source reputation can lead to contracting and consulting offers.

Conclusion

With the demand for data skills remaining robust even during economic uncertainty, data professionals have an advantage when it comes to high paying side hustles. Whether you want to consult, teach online courses, do freelance analytics projects, write technical content, or participate in data science competitions, there are plenty of ways to profit from your expertise.

With the flexibility and reward potential of side hustles, data scientists and technical professionals can thrive financially even when corporate salaries stagnate. Rather than being limited to traditional employment, embrace entrepreneurial ideas that allow you to monetize your skills.
Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master's degree in Technology Management and a bachelor's degree in Telecommunication Engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.

More On This Topic

  • The High Paying Side Hustles for Data Scientists
  • KDnuggets™ News 22:n04, Jan 26: The High Paying Side Hustles for Data…
  • 6 side hustles for an aspiring data scientist
  • 6 Highest Paying Companies for Data Scientists
  • High-Fidelity Synthetic Data for Data Engineers and Data Scientists Alike
  • How CoRise Helped Ben Wilson Land a New Job as a Analytics Engineer (and a…

8 ways to reduce ChatGPT hallucinations

AI hallucinations

"Liar, liar. Pants on fire."

One of the most troubling aspects of working with large language model chat AIs is their tendency to make stuff up, fabricate answers, and otherwise present as fact information that is completely wrong.

For example, in an article about using ChatGPT to write code, I showed how ChatGPT incorporated the following URL into the code:

https://www.reuters.com/business/retail-consumer/teslas-musk-says-fremont-california-factory-may-be-sold-chip-shortage-bites-2022-03-18/

It looks legitimate, doesn't it? After all, Reuters is a very credible news source. It looks like it's an article about Tesla selling a factory, written in March of 2022. But, of course, ChatGPT doesn't model data from March of 2022, and the factory wasn't being sold. It's a complete fabrication made up out of the ether by ChatGPT. That link doesn't go anywhere. 404 to the max, baby.

Also: How to use Claude AI (and how it's different from ChatGPT)

That ChatGPT "hallucinates" is a known and common problem. OpenAI (the makers of ChatGPT) co-founder John Schulman says, "Our biggest concern was around factuality, because the model likes to fabricate things."

But what if you want to use ChatGPT and get good-quality answers? It is possible. In this article, I'll show you eight ways to reduce hallucinations. It's all about how you ask your questions.

Also: OK, so ChatGPT just debugged my code. For real

In each of these best practices, I'm including five examples that show how not to use the AI. If you paste these as-is into a chatbot, you'll probably get a caution that they contain impossible requests. I'm using these as examples. The key is to avoid accidentally using these hallucination-prompting request styles embedded in more realistic questions.

Let's get started.

1. Avoid ambiguity and vagueness

When prompting an AI, it's best to be clear and precise. Prompts that are vague, ambiguous, or do not provide sufficient detail to be effective give the AI room to confabulate in an attempt to fill in the details you left out.

Also: How to use Bing Image Creator (and why it's better than ever)

Here are some examples of prompts that are too ambiguous and might result in an inaccurate or fabricated result:

  • Discuss the event that took place last year.
  • Describe the impact of that policy on people.
  • Outline the development of technology in the region.
  • Describe the effects of the incident on the community.
  • Explain the implications of the experiment conducted recently.

Keep in mind that most prompts will likely violate more than one of the eight factors described in this article. While the examples shown here are intended for illustration, an actual prompt you write may have ambiguity buried among other details. Evaluate your prompts with care, making sure to pay special attention to errors like those shown here.

2. Avoid merging unrelated concepts

Prompts that merge unrelated concepts, that combine incongruent concepts in a single prompt, or have no direct relationship or correlation, may well induce the AI to fabricate a response that implies the unrelated concepts are, in fact, related.

Here are some examples:

  • Discuss the impact of ocean currents on internet data transfer speeds across continents.
  • Describe the relationship between agricultural crop yields and advancements in computer graphics technology.
  • Detail how variations in bird migration patterns affect global e-commerce trends.
  • Explain the correlation between the fermentation process in winemaking and the development of electric vehicle batteries.
  • Describe how different cloud formations in the sky impact the performance of stock trading algorithms.

Remember that the AI doesn't actually know anything about our world. It will attempt to fit what it's being asked to do into its model, and if it can't fit it using actual facts, it will attempt to interpolate, providing fabrications or hallucinations where it needs to fill in the blanks.

3. Avoid describing impossible scenarios

Within your prompts, be sure to use scenarios that are practical and real. Scenarios that are physically or logically impossible, in turn, induce hallucinations.

Also: How to create your own comic books with AI

Here are some examples:

  • Explain the physics of environmental conditions where water flows upward and fire burns downwards.
  • Explain the process by which plants utilize gamma radiation for photosynthesis during nighttime.
  • Describe the mechanism that enables humans to harness gravitational pull for unlimited energy generation.
  • Discuss the development of technology that allows data to be transmitted faster than the speed of light.
  • Detail the scientific principles that allow certain materials to decrease in temperature when heated.

If the AI doesn't detect the impossibility of such a scenario, it will build upon it. But if the foundation is impossible, the response will also be impossible.

4. Avoid using fictional or fantastical entities

Within your prompts, it's important to give the AI a foundation that's as grounded in fact as possible. Unless you're purposely playing with fictional concepts (as I did with asking ChatGPT to write a Star Trek story), stay firmly grounded in reality.

Also: Can generative AI solve computer science's greatest unsolved problem?

While fictional entities, objects, and concepts might help you explain something, they could lead the chatbot astray. Here are a number of examples of what not to do:

  • Discuss the economic impact of the discovery of vibranium, a metal that absorbs kinetic energy, on the global manufacturing industry.
  • Explain the role of flux capacitors, devices that enable time travel, in shaping historical events and preventing conflicts.
  • Describe the environmental implications of utilizing the Philosopher's Stone, which can transmute substances, in waste management and recycling processes.
  • Detail the impact of the existence of Middle Earth on geopolitical relations and global trade routes.
  • Explain how the use of Star Trek's transporter technology has revolutionized global travel and impacted international tourism.

As you can see, the fantastical concepts might be fun to play with. But using them in serious prompts could well cause the AI to return wildly fabricated answers.

5. Avoid contradicting known facts

Don't use prompts that contain statements that contradict well-established facts or truths, because those contradictions can open the door to confabulation and hallucinations.

Here are some examples of that practice:

  • Discuss the impact of the Earth being the center of the universe on modern astrophysics and space exploration.
  • Detail the effects of a flat Earth on global climate patterns and weather phenomena.
  • Explain how the rejection of germ theory, the concept that diseases are caused by microorganisms, has shaped modern medicine and hygiene practices.
  • Describe the process by which heavier-than-air objects naturally float upwards, defying gravitational pull.
  • Explain how the concept of vitalism, the belief in a life force distinct from biochemical actions, is utilized in contemporary medical treatments.

These ideas are also fun to play with, but if you're looking for reliable results from the large language model, stick to commonly accepted facts and avoid ideas that might be misinterpreted.

6. Avoid misusing scientific terms

When prompting, be careful about using scientific terms, especially if you're not precisely sure what they mean. If you use prompts that misapply scientific terms or concepts in a way that sounds plausible but are scientifically inaccurate, the language model is likely to try to find a way to make them work. The result: fabricated answers.

Also: Generative AI will far surpass what ChatGPT can do. Here's how the tech advances

Here are five examples of what I mean:

  • Explain how utilizing Heisenberg's uncertainty principle in traffic engineering can minimize road accidents by predicting vehicle positions.
  • Describe the role of the placebo effect in enhancing the nutritional value of food without altering its physical composition.
  • Outline the process of using quantum entanglement to enable instantaneous data transfer between conventional computers.
  • Detail the implications of applying the observer effect, the theory that simply observing a situation alters its outcome, in improving sports coaching strategies.
  • Explain how the concept of dark matter is applied in lighting technologies to reduce energy consumption in urban areas.

See how some of these things sound plausible? In most cases, the AI will probably tell you that the ideas are speculative, and the answer being provided is merely an exercise. But if you aren't really careful about wording, the AI might be fooled into treating these garbage-in terms as real, and the result will be very confidently presented garbage-out.

7. Avoid blending different realities

As someone who enjoys science fiction, I enjoy speculative scenarios and alternative reality stories. But when trying to get clear answers from an AI, be careful about mixing elements from different realities, timelines, or universes in a way that sounds plausible but are just not possible.

Here are some examples:

  • Discuss the impact of the invention of the internet during the Renaissance period on art and scientific discovery.
  • Explain how the collaboration between Nikola Tesla and modern-day artificial intelligence researchers shaped the development of autonomous technologies.
  • Describe the implications of utilizing World War II-era cryptography techniques to secure contemporary digital communications.
  • Outline the development of space travel technologies during Ancient Egyptian civilization and its impact on pyramid construction.
  • Discuss how the introduction of modern electric vehicles in the 1920s would have influenced urban development and global oil markets.

One reason to be careful about these sorts of prompts is you might not have the knowledge to validate the responses. Take a look at the last example, electric cars in the 1920s. Most folks might laugh off the idea, knowing electric cars are a modern innovation. But that would actually be wrong.

Also: ChatGPT vs. Bing Chat vs. Google Bard: Which is the best AI chatbot?

Some of the first electric vehicles were actually invented back in the 1830s. Yep, quite a bit of time before the internal combustion engine. That's right, folks. Keep coming back to ZDNET. Not only do we provide hands-on tips for using AI, but we'll blow your mind with an impromptu tech history lesson!

8. Avoid assigning uncharacteristic properties

We'll wrap up our list of avoidance practices with this one: avoid crafting prompts that assign properties or characteristics to entities that they do not possess in a way that sounds plausible but are scientifically inaccurate.

Here are some examples:

  • Explain how the magnetic fields generated by butterfly wings influence global weather patterns.
  • Describe the process by which whales utilize echolocation to detect pollutants in ocean water.
  • Outline the role of bioluminescent trees in reducing the need for street lighting in urban areas.
  • Discuss the role of the reflective surfaces of oceans in redirecting sunlight to enhance agricultural productivity in specific regions.
  • Explain how the electrical conductivity of wood is utilized in creating eco-friendly electronic devices.

The idea here is you're using a property of an object, like a color or a texture, and then relating it to some other object that doesn't have that property.

Final thoughts

Some of these precautions can stack. Take, for example, this prompt:

How do I keep the hair on my mouse clean?

This is where context can be king. Hair is certainly a property of living creatures but is not normally the property of a computer mouse. But it is a property of a pet mouse. In this one prompt, we're violating the "avoid ambiguity" rule because we didn't specify what kind of mouse and, possibly, violating the "uncharacteristic properties" caution if we're talking about hair on a computer mouse.

Another thing to be concerned about is how prompting and "facts" fit into an overall worldview. All the AI companies (and many tech companies) are dealing with this issue.

Also: DALL-E 3 is now available for free in Bing Chat

That's because, in modern society, we have a bit of a problem with facts. Depending on cultural background, political affiliation, religious beliefs, or merely upbringing, what is considered absolute fact by one person may be considered fantasy by another. Keep in mind that those perspectives may also color the results of the AI, and try to avoid contested topics if you're trying to get reliable answers from the machine.

Overall, though, if you follow these guidelines and avoid constructing prompts that could confuse the AI, you stand a better chance of reducing hallucinations.

Let us know if you've tried out any of these tactics (or have others). Have any worked for you? Did ChatGPT ever hallucinate for you in any spectacular or interesting ways? Let us know in the comments below.

You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter on Substack, and follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.

More on AI tools

How Israel is Using AI to Defuse Hamas Militants 

Israel has declared a state of war, after Hamas, a militant group which controls Gaza, attacked the border towns of Israel in the early hours of Saturday. There are more than 100,000 people displaced in Gaza as Israel continues to retaliate on Hamas’ attack.

Credits : CNN

The casualties could have been significantly higher if the IDF had not deployed their ‘Iron Dome’ technology which is protecting the Israeli skies.

Israel’s Iron Dome System

Hamas, which is a Palestinian militant group, operating mainly from the Gaza Strip, fired at least 3,000 rockets in Israel on Saturday. The IDF said that it caused little or no harm as their Iron Dome air defence system intercepted almost 90% of the rockets being fired.

The main reason why Iron Dome was created was to intercept rockets and mortars being shot at Israel, at a close distance.

Iron Dome operates using radar technology. It detects an enemy incoming rocket, from up to 70 kms away; using predictive analytics and machine learning–works with data such as speed, weather and size) to calculate the trajectory and point of impact.

Credits : Bloomberg

The control centre analyses that information and subsequently launches a counter-missile to nullify the enemy rocket. If the enemy rocket is headed to terrains where it’s unpopulated and no damage is estimated, then the system doesn’t do anything.

How Israel has integrated AI into military

The Israeli Defence Forces are using AI to organise logistics during wartime, reduce time taken for decision-making and selecting targets. They use a system of AI recommendations which churches large amounts of data and selects targets for air strikes. After this, raids and logistics are quickly assembled using another AI software called ‘Fire Factory’–which uses data about targets to calculate ammunition loads, assign targets to fighter jets and military drones. It also prepares a schedule, saving loads of time and potentially lives.

All AI suggestions are overseen and corrected if necessary by human operators. Research agrees that the use of AI in modern warfare can reduce casualties, both from military as when as civilians.

This AI system is trained on thousands of hours of footage to identify people and objects. It interprets large amounts of drone footage, satellite imagery, electronic signals and online communications, according to Bloomberg.

In 2021, Israel had used the AI-software to identify rocket launchpads and deploy drone swarms. They also use AI to analyse satellite imagery. The IDF using AI, had raided weapon shipments in Syria and Lebanon which were to be delivered to Iran backed militants like Hezbollah.

“What used to take hours now takes minutes, with a few more minutes for human review,” said Col. Uri, who heads the army’s digital transformation unit to Bloomberg.

Israel is also using Facial Recognition technology to scaledown on Palestinians. The army had such extensive images, that one former soldier described it as the army’s secret “Facebook for Palestinians” , according to the Washington Post.

The operation is being conducted for over 2 years now called “Blue Wolf”. This is an effort by Israel to defend against terrorists.

Who’s Funding it?

Coming back to Iron Dome, it was initially developed without the help of the U.S, but from 2011, they started to support the program financially. Now, some of the missiles are made by the U.S. America is a huge support for Israel. They are signed a contract, where U.S will provide $38 billion in total from 2019 to 2028

Israel wants to become an AI “superpower” and to advance in autonomous warfare and combat decision-making, said the Director of Israel’s defence ministry.

“There are those who see AI as the next revolution in changing the face of warfare in the battlefield,” said Eyal Zamir, a retired army general, at the Herzliya Conference, an annual international security forum.

The details of how much the country wants to spend on this AI-revolution remains undisclosed.

The post How Israel is Using AI to Defuse Hamas Militants appeared first on Analytics India Magazine.

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas
Image by Freepik

This comprehensive article will discuss time-based data visualization using Python with the Pandas library. As you know, time-series data is a treasure trove of insights, and with the skillful resampling technique, you can transform raw temporal data into visually compelling narratives. Whether you're a data enthusiast, scientist, analyst, or just curious about unraveling the stories hidden within time-based data, this article help you with the knowledge and tools to revamp your data visualization skills. So, let’s start discussing the Pandas resampling techniques and turn data into informative and captivating temporal masterpieces.

Why Data Resampling?

While working with time-based data visualization, Data resampling is crucial and very useful. It allows you to control the granularity of the data to extract meaningful insights and create visually compelling representations to understand it better. In the below picture, you can observe that you can either upsample or downsample your time series data in terms of frequencies based on your requirements.

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas
Image from SQLRelease

Basically, the two primary purposes of data resampling are mentioned below:

  1. Granularity Adjustment: Collecting the big data allows you to change the time intervals at which data points are collected or aggregated. You can get only the vital information instead of getting the noise. This can help you remove the noisy data, which converts the data to more manageable for visualization.
  1. Alignment: It also helps align data from multiple sources with different time intervals, ensuring consistency when creating visualizations or conducting analyses.

For Example,

Suppose you have daily stock price data for a particular company that you are getting from a stock exchange, and you aim to visualize the long-term trends without including the noisy data points in your analysis. So, to do this, you can resample this daily data to a monthly frequency by taking the average closing price for each month, and as a result, the size of the data for visualization purpose decrease, and your analysis can provide better insights.

import pandas as pd  # Sample daily stock price data  data = {  'Date': pd.date_range(start='2023-01-01', periods=365, freq='D'),  'StockPrice': [100 + i + 10 * (i % 7) for i in range(365)]  }  df = pd.DataFrame(data)  # Resample to monthly frequency  monthly_data = df.resample('M', on='Date').mean()  print(monthly_data.head())

In the above example, you have observed that we have resampled the daily data into monthly intervals and calculated the mean closing price for each month, due to which you got the smoother, less noisy representation of the stock price data, making it easier to identify long-term trends and patterns for decision making.

Choosing the Right Resampling Frequency

When working with time-series data, the main parameter for resampling is the frequency, which you must select correctly to get insightful and practical visualizations. Basically, there is a tradeoff between granularity, which implies how detailed the data is, and clarity, which means how well the data patterns are revealed.

For Example,

Imagine you have temperature data recorded every minute for a year. Suppose you have to visualize the annual temperature trend; using minute-level data would result in an excessively dense and cluttered plot. On the other hand, if you aggregate the data to yearly averages, you might lose valuable information.

# Sample minute-level temperature data  data = {      'Timestamp': pd.date_range(start='2023-01-01', periods=525600, freq='T'),      'Temperature': [20 + 10 * (i % 1440) / 1440 for i in range(525600)]  }    df = pd.DataFrame(data)    # Resample to different frequencies  daily_avg = df.resample('D', on='Timestamp').mean()  monthly_avg = df.resample('M', on='Timestamp').mean()  yearly_avg = df.resample('Y', on='Timestamp').mean()    print(daily_avg.head())  print(monthly_avg.head())  print(yearly_avg.head())

In this example, we resample the minute-level temperature data into daily, monthly, and yearly averages. Depending on your analytical or visualization goals, you can choose the level of detail that best serves your purpose. Daily averages reveal daily temperature patterns, while yearly averages provide a high-level overview of annual trends.

By selecting the optimal resampling frequency, you can balance the amount of data detail with the clarity of your visualizations, ensuring your audience can easily discern the patterns and insights you want to convey.

Aggregation Method and Techniques

When working with time-based data, it's essential to understand various aggregation methods and techniques. These methods allow you to effectively summarize and analyze your data, revealing different aspects of your time-based information. Standard aggregation methods include calculating sums and means or applying custom functions.

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas
Image from TowardsDataScience

For Example,

Consider you have a dataset containing daily sales data for a retail store over a year. You want to analyze the yearly revenue trend. To do this, you can use aggregation methods to calculate the total sales for each month and year.

# Sample daily sales data  data = {  'Date': pd.date_range(start='2023-01-01', periods=365, freq='D'),  'Sales': [1000 + i * 10 + 5 * (i % 30) for i in range(365)]  }  df = pd.DataFrame(data)    # Calculate monthly and yearly sales with the aggregation method  monthly_totals = df.resample('M', on='Date').sum()  yearly_totals = df.resample('Y', on='Date').sum()    print(monthly_totals.head())  print(yearly_totals.head())

In this example, we resample the daily sales data into monthly and yearly totals using the sum() aggregation method. By doing this, you can analyze the sales trend at different levels of granularity. Monthly totals provide insights into seasonal variations, while yearly totals give a high-level overview of the annual performance.

Depending on your specific analysis requirements, you can also use other aggregation methods like calculating means and medians or applying custom functions depending on the dataset distribution, which is meaningful according to the problem. These methods allow you to extract valuable insights from your time-based data by summarizing it in a way that makes sense for your analysis or visualization goals.

Handling Missing Data

Handling missing data is a critical aspect of working with time series, ensuring that your visualizations and analyses remain accurate and informative even when dealing with gaps in your data.

For Example,

Imagine you're working with a historical temperature dataset, but some days have missing temperature readings due to equipment malfunctions or data collection errors. You must handle these missing values to create meaningful visualizations and maintain data integrity.

# Sample temperature data with missing values  data = {      'Date': pd.date_range(start='2023-01-01', periods=365, freq='D'),      'Temperature': [25 + np.random.randn() * 5 if np.random.rand() > 0.2 else np.nan for _ in range(365)]  }  df = pd.DataFrame(data)    # Forward-fill missing values (fill with the previous day's temperature)  df['Temperature'].fillna(method='ffill', inplace=True)    # Visualize the temperature data  import matplotlib.pyplot as plt  plt.figure(figsize=(12, 6))  plt.plot(df['Date'], df['Temperature'], label='Temperature', color='blue')  plt.title('Daily Temperature Over Time')  plt.xlabel('Date')  plt.ylabel('Temperature (°C)')  plt.grid(True)  plt.show()

Output:

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas
Image by Author

In the above example, you can see that firstly, we have simulated the missing temperature values (about 20% of the data) and then used the forward-fill (ffill) method to fill in the gaps, which means that the missing values are replaced with the temperature from the previous day.

Therefore, handling the missing data ensures that your visualizations accurately represent the underlying trends and patterns in the time series, preventing gaps from distorting your insights or misleading your audience. Various strategies, such as interpolation or backward-filling, can be employed based on the nature of the data and the research question.

Visualizing Trends and Patterns

Data resampling in pandas allows you to visualize trends and patterns in sequential or time-based data, which further helps you to collect insights and effectively communicate the results to others. As a result, you can find clear and informative visual representations of your data to highlight the different components, including trends, seasonality, and irregular patterns (possibly the noise in the data)

For Example,

Suppose you have a dataset containing daily website traffic data collected over the past years. You aim to visualize the overall traffic trend in the subsequent years, identify any seasonal patterns, and spot irregular spikes or dips in traffic.

# Sample daily website traffic data  data = {  'Date': pd.date_range(start='2019-01-01', periods=1095, freq='D'),  'Visitors': [500 + 10 * ((i % 365) - 180) + 50 * (i % 30) for i in range(1095)]  }  df = pd.DataFrame(data)    # Create a line plot to visualize the trend  plt.figure(figsize=(12, 6))  plt.plot(df['Date'], df['Visitors'], label='Daily Visitors', color='blue')  plt.title('Website Traffic Over Time')  plt.xlabel('Date')  plt.ylabel('Visitors')  plt.grid(True)    # Add seasonal decomposition plot  from statsmodels.tsa.seasonal import seasonal_decompose  result = seasonal_decompose(df['Visitors'], model='additive', freq=365)  result.plot()  plt.show()

Output:

Revamping Data Visualization: Mastering Time-Based Resampling in Pandas
Image by Author

In the above example, we have first created a line plot to visualize the daily website traffic trend over time. This plot describes the overall growth and any irregular patterns in the dataset. Also, to decompose the data into different components, we use the seasonal decomposition technique from the statsmodels library, including trend, seasonality, and residual components.

This way, you can effectively communicate the website's traffic trends, seasonality, and anomalies to stakeholders, which enhances your ability to derive important insights from time-based data and convert it into data-driven decisions.

Wrapping it Up

Colab Notebook link: https://colab.research.google.com/drive/19oM7NMdzRgQrEDfRsGhMavSvcHx79VDK#scrollTo=nHg3oSjPfS-Y

In this article, we discussed the time-based resampling of data in Python. So, to conclude our session, let’s summarize the important points covered in this article:

  1. Time-based resampling is a powerful technique for transforming and summarizing time-series data to get better insights for decision-making.
  2. Careful selection of resampling frequency is essential to balance granularity and clarity in data visualization.
  3. Aggregation methods like sum, mean, and custom functions help reveal different aspects of time-based data.
  4. Effective visualization techniques aid in identifying trends, seasonality, and irregular patterns, facilitating clear communication of findings.
  5. Real-world use cases in finance, weather forecasting, and social media analytics demonstrate the wide-ranging impact of time-based resampling.

Aryan Garg is a B.Tech. Electrical Engineering student, currently in the final year of his undergrad. His interest lies in the field of Web Development and Machine Learning. He have pursued this interest and am eager to work more in these directions.

More On This Topic

  • The Role of Resampling Techniques in Data Science
  • Deep Learning-based Real-time Video Processing
  • Multivariate Time Series Analysis with an LSTM based RNN
  • KDnuggets News, November 9: 7 Tips To Produce Readable Data Science Code •…
  • 30 Resources for Mastering Data Visualization
  • Cloud Based Web Scraping for Big Data Applications

How BharatPe is Harnessing Generative AI for the Future of Finance

The finance sector has been grappling with significant data security issues, ethical concerns, and the complexities of complying with stringent regulations. AI offers solutions to these problems, but the industry has been cautious about adopting AI due to the sensitive nature of the data involved, potential ethical issues in AI-generated financial advice, and the overall challenge of navigating regulatory frameworks.

However, there is a gradual shift occurring.

When it comes to sifting through vast datasets, discerning current trends, and projecting potential shifts in market dynamics and customer behaviour, AI plays a crucial role. Fintech companies are discovering practical use cases for AI in these areas.

“For a fintech entity like BharatPe, AI is instrumental in running credit-scoring models that predict credit eligibility, especially for people with limited or no credit history,” said Ritesh Srivastava, Chief Data Scientist at BharatPe. He also emphasised how these algorithms work into unconventional data, including expenditure on necessities versus luxuries, and may even analyse social media activity or online shopping habits to build a comprehensive profile of a borrower’s financial behaviour.

This comprehensive understanding of financial behaviour helps in making informed decisions about loan offers and determining the appropriate loan amount. Additionally, for BharatPe, AI facilitates the delivery of personalised offers and communication to customers, aligning with the contemporary demand for individualised services in the digital age. This customer-centric approach represents a significant departure from the traditional product-centric approach of the past.

Srivastava has been with BharatPe since last year. The IIM-Bengaluru alumni have also served in important positions as Associate Director of data science at pharma company Novartis and Head of data and analytics at travel platform Yatra.

Inside BharatPe’s Data Science Team

BharatPe’s AI and analytics team, comprising 30 professionals, including 10 specialised in AI and ML, is pivotal in the organisation, contributing significantly to innovation and data-driven decision-making, involving close cooperation with data engineers, scientists, and domain experts, ensuring technical excellence aligns with business goals.

“We employ robust AI algorithms to analyse customer behaviour, identify patterns, and generate actionable insights for strategic decision-making, facilitating a deep understanding of customer purchasing patterns, identifying cross-selling and upselling opportunities and personalising our offerings,” said Srivastava.

BharatPe’s AI/ML-driven fraud and risk engine is adept at real-time prevention of anomalies and social engineering. Leveraging a diverse set of risk factors and social-behavioural parameters, the engine adapts autonomously to evolving threats. Real-time feedback enhances the engine’s deep learning capabilities, enabling it to recognise and fight historical patterns in future transactions.

He further added that agile development practices along with collaboration with product and engineering teams ensure scalability and robustness, vital for real-world conditions serving millions of users.

“To stay ahead in this evolving landscape, we’ve established a dedicated research team focused on AI, investing in continuous upskilling of our tech teams,” he added.

Decoding BharatPe’s Generative AI Progress

In terms of how generative AI has transformed operations at BharatPe, Srivastava said that it improved their capabilities by identifying undiscovered trends and influencing our business landscape.

“Its adoption is expected to bring notable transformations to various departments in our company, giving rise to specialised units focused on generative AI,” said Srivastava. These units will play a vital role in restructuring procedures for customer service products and addressing data gaps through the collection of unstructured data from diverse sources, with the added benefit of minimising biases in data analysis.

“Leveraging cloud-based infrastructure facilitates the quick scalability of systems to accommodate business growth so the integration of ML outcomes directly informs real-time decision-making processes,” he added.

The team is currently experimenting with different generative AI tools for enhanced market analysis and a deeper understanding of customer behaviour. These tools play a crucial role in market forecasting, enabling proactive anticipation of trends and informed planning for new products or technologies.

Talking about partnerships, even though Srivastava could not take any names due to confidentiality, he said that they are actively pursuing strategic collaborations and acquisitions with institutions and tech firms in the generative AI space to enhance our data analytics capabilities.

The post How BharatPe is Harnessing Generative AI for the Future of Finance appeared first on Analytics India Magazine.

MongoDB Announced New Generative AI Features for Developers

At MongoDB.local in London, the non-relational database giant has introduced a set of generative AI features across various tools to streamline and enhance application development and modernisation.

The MongoDB Relational Migrator now includes AI-powered capabilities that significantly improve the migration process from legacy database technologies to MongoDB Atlas. This tool automates the conversion of SQL queries and stored procedures in legacy applications to development-ready MongoDB Query API syntax, allowing organisations to accelerate their migration efforts without requiring extensive knowledge of MongoDB Query Syntax API.

In MongoDB Compass, the data interaction tool, developers can now leverage natural language to swiftly generate executable MongoDB Query API syntax. By entering commands such as ‘Filter pizza orders by size, group the remaining documents by pizza name, and calculate the total quantity,’ developers receive suggested code to execute the necessary aggregation pipeline stages. This natural language capability enables developers to focus more on shipping data-driven applications, reducing the manual effort required for complex queries and aggregations.

Data visualisation tool MongoDB Atlas Charts has integrated AI-powered capabilities to facilitate the creation of visualisations using natural language commands. Developers can input queries like ‘Show me a comparison of annual revenue by country and product,’ and MongoDB Atlas Charts will swiftly generate the requested visualization. The familiar drag-and-drop interface then allows for further refinement and customization, enabling developers to efficiently create, share, and embed visualizations.

Additionally, MongoDB Documentation now features an AI-powered chatbot that provides quick and intuitive answers to developers’ questions. Developers can ask about MongoDB’s products and services, troubleshoot issues during software development, and receive step-by-step instructions, example code, and links to references. The chatbot, an open-source project utilising MongoDB Atlas Vector Search, facilitates information retrieval with context, allowing developers to build and deploy their own chatbots for various use cases. This integration of generative AI features across MongoDB tools aims to reduce the time and effort spent on undifferentiated tasks, allowing developers to focus on innovation and creating exceptional end-user experiences.

Two weeks ago, the NY-based company introduced features in MongoDB Atlas Vector Search that benefit generative AI application development. These features enhance information LLMs by expanding query capabilities and facilitating a dedicated data aggregation stage, reducing inaccuracies. The platform also accelerates data indexing for generative AI applications by simplifying the indexing process for operational data, metadata, and vector data, thereby speeding up the development of AI-powered applications.

Read more: Is MongoDB Vector Search the Panacea for all LLM Problems?

The post MongoDB Announced New Generative AI Features for Developers appeared first on Analytics India Magazine.