Рубрика: AI

Apple is bringing generative AI to Siri and throughout iOS 18, iPad OS 18 and macOS Sequoia, the company announced at the Apple Worldwide Developers Conference on June 10.

As was rumored, Apple partnered with OpenAI to link ChatGPT to iOS, iPadOS and macOS. However, Apple Intelligence will be an independent generative AI service hosted on its own servers.

Adding generative capabilities to Siri has the advantage of familiarity, and partnering with OpenAI means Apple won’t have to do all of the work itself. Apple is betting on generative AI being a seamless addition to the way the Apple ecosystem may already organize a consumer’s or professional’s life.

Apple Intelligence will be available in beta this fall in iOS 18, iPadOS 18 and macOS Sequoia. It will work on iPhone 15 Pro, iPhone 15 Pro Max and some other current iPad and Mac devices with M1 or newer chips. ChatGPT assistance for Apple Intelligence will follow later in 2024.

Siri will respond to natural language

The popular voice assistant is catching up to the generative AI world with Apple Intelligence, which enables Siri to:

• Understand more natural-sounding questions and speech patterns, including understanding pauses or corrections.
• Remember previous conversations and references.
• Answer questions about Apple devices.
• Take actions inside apps on your behalf.

You can now type to talk to Siri, double tapping the bottom of the screen to make natural-language requests.

Apple didn’t show flashy demos like OpenAI and Google did, but they may be betting on more consumers finding uses for generative AI through what they already do with Siri.

Enhancements to Siri with Apple Intelligence will roll out over the course of the next year. Developers will be able to define how Siri with Apple Intelligence will be able to interact with their apps.

What is Apple Intelligence?

During the AI reveal at WWDC, Apple put the emphasis on “personal intelligence” and privacy. Most of the generative AI processes used for Apple Intelligence will be performed on-device on the A17 or M-series chips and in a proprietary cloud.

Craig Federighi, Apple senior vice president of software engineering, emphasized personal context in the WWDC keynote.

“We’re tremendously excited about the power of generative models … But these tools know very little about you and your needs,” he said.

In response, Apple Intelligence on iOS 18, iPad 18 and macOS Sequoia will use AI for “personal context” to make day-to-day tasks faster. The iPhone can prioritize and summarize notifications with the “priority notifications” feature, selecting the most important ones first.

SEE: Concerns about ultra-powerful artificial intelligence lead some within Silicon Valley to seek a ‘Right to Warn’ for AI project whistleblowers.

What else does understanding personal context mean? Apple Intelligence can determine what data you think is most relevant and reference related content. For example, if your meeting is rescheduled, you can check whether the new time will prevent you from getting to your daughter’s school event. Answering this question requires Apple Intelligence to search and reason across your personal and work calendars, email and Apple Maps.

New writing tools with generative AI will be available in Mail, Notes, Pages, Keynote and third party apps. Apple Intelligence can write and summarize emails, too.

Apple Intelligence can take action between apps to carry out tasks on your behalf. For example, you can ask it to take action based on natural questions like “Play the podcast that my wife sent the other day.”

Apple Intelligence lets consumers create custom emojis easily in apps like Messages. Notes, Pages, Keynote and Freeform will all gain generative AI images. Apple intelligence is able to identify and recreate pictures of people in your photo library,” such as creating an image of them for their birthday.

Private Cloud Compute

Sometimes, Apple’s AI processes will need to be performed on external servers. To hopefully prevent privacy and security problems around this, Apple will run these processes on Private Cloud Compute, a home for Apple Intelligence that are still thoroughly within the Apple ecosystem and use Apple’s own Swift programming language. Even Apple can’t see that data, the company claims.

“You should not have to hand over all the details of your life to be warehoused and analyzed in someone’s AI cloud,” said Federighi.

ChatGPT will be integrated with Siri

In addition to Apple Intelligence, Apple is partnering with OpenAI to bring ChatGPT and GPT-4o to Siri, which will be used when referencing current information or gathering information that isn’t on your devices.

Siri will ask for permission to reach out to ChatGPT. You can do this for free or connect to your personal ChatGPT subscription. This integration is coming later this year. Plus, Apple plans to add support for other AI models.

Where Apple’s AI fits in to the competition

Apple has seemingly chosen to stay out of the generative AI chatbot race for the last few years, although it published research on running large language models on mobile devices in December 2023. The relatively late entry means Apple has had time to watch the rest of the industry try to find practical use cases for generative AI. It’s also seen Microsoft switch its Recall AI feature from default to opt-in after security concerns.

Will Apple face the same backlash and security digging as Microsoft’s Recall? And is Apple’s ecosystem so powerful that its email summarization and other Apple Intelligence capabilities will prove as game-changing as AI companies predict generative AI content to be? Is Apple playing it safe when it comes to AI? It’s too early to say, but time will tell whether Apple Intelligence might be opt-in — or transformative — going forward.

Apple is expected to have one of its most groundbreaking Worldwide Developers Conference (WWDC) keynotes today, as the company plans to add major artificial intelligence (AI) features to its operating systems.

But rather than unveiling a slew of flashy generative AI features to knock your socks off, Apple is expected to focus on incorporating AI into its apps to simplify users' daily tasks. And it'll categorize such features under the name "Apple Intelligence."

Also: The best Apple deals of June 2024: iPhones, Apple Watches, iPads, and more

According to Bloomberg, Apple is branding its AI features under Apple Intelligence — and we didn't miss the snarky wordplay. Apple Intelligence will include the latest AI features coming to its operating systems, including iOS, iPadOS, MacOS, and WatchOS.

Apple Intelligence focuses on broad-appeal AI features rather than advanced image and video generation. To do this, the company developed in-house AI models and partnered with OpenAI to power a chatbot that will work similarly to ChatGPT.

Also: Apple to give Control Center its most useful customization feature ever with iOS 18

Some of the biggest AI features we're expecting with Apple Intelligence include:

• Improved photo editing, like object removal, with AI in Photos.
• Greater Siri control over apps and actions, including asking Siri to delete emails or edit photos.
• AI generation of custom emojis based on text prompts.
• Generating quick recaps of notes, text messaging threads, emails, and more texts.
• Automatically suggesting responses for emails and messages.
• An improved Mail app that can categorize emails and generate messages.
• Automatic transcription of voice memos.
• AI enhancements for Xcode to auto-complete code.

Aside from these AI features, Bloomberg reports that iOS 18 will include new customizable icons and interface updates for Control Center, Settings, and Messages. Apple is also expected to launch a new Passwords app to replace the iCloud Keychain and give users a more user-friendly option, similar to 1Password and LastPass.

Also: Apple to unveil 'Passwords' manager app at WWDC 2024: What it is and how it works

On-device vs. cloud-based

Though Apple was rumored to work on different ways to keep its AI running strictly on-device for security and privacy, Apple Intelligence is expected to rely on the cloud for at least some tasks. This will depend on device complexity, resource availability, data privacy considerations, and latency requirements.

Essentially, if a task is simple enough to be processed locally, leveraging the device's processing power and battery life, and requires immediate results, it is more likely to be handled on-device. Tasks involving sensitive data could also prioritize on-device processing, as Apple tries to prioritize data privacy.

Also: ChatGPT privacy tips: Two important ways to limit the data you share with OpenAI

In turn, cloud-based AI processing requires sending data from the device to remote servers that can handle complex or computationally heavy tasks. In Apple's case, tasks requiring processing large amounts of data or updated models could include advanced natural language processing (NLP), intricate analysis, and complex image and video generation.

Depending on its complexity and system requirements, an algorithm will determine whether a task requiring AI should be processed on-device or offloaded to the cloud. Simpler tasks like a Siri request and other basic NLP tasks can be processed on-device. More complex tasks, like generating a detailed summary of a large document, will be sent to the cloud, where more robust processing can occur.

Technical requirements for Apple Intelligence

According to Bloomberg, Apple's new AI features will be compatible with the latest Apple devices, including iPhone 15 Pro or newer models, which run on an A17 Pro chip, and iPads and Macs with an M1 chip or newer. While these AI features may help drive sales of new iPhones and Macs, as a current iPhone 14 Pro Max owner, I hope that at least some will trickle down to older iPhone models. We'll know what the official compatibility list is come WWDC.

Also: Apple Photos app is getting an AI-powered editing feature to wipe out photobombers

During the event, Apple is expected to highlight new security measures for running AI tasks, including chip-based security in data centers for cloud-based processing. It will also reiterate its commitment not to build user profiles based on consumer data.

Perhaps most importantly, users can opt-in for Apple Intelligence features, which will be introduced as beta versions as Apple works to improve its AI capabilities over time.

Featured

Soon, Siri will be able to tap ChatGPT for "expertise" where it might be helpful, Apple says.

© 2024 TechCrunch. All rights reserved. For personal use only.

The Falcon family of large language models (LLMs) – developed by the Technology Innovation Institute (TII) in Abu Dhabi – demonstrate impressive capabilities. Falcon LLMs span across a wide variety of parameter sizes, as well as two generations:

• Falcon 1.3B with 1.3 billion parameters
• Falcon 7.5B with 7.5 billion parameters
• Falcon 40B with 40 billion parameters
• Falcon 180B with 180 billion parameters
• Falcon 180B Instruct with 180 billion parameters

Falcon 180B is one of the larger LLMs in the industry and was trained on a dataset of over 3.5 trillion tokens from publicly available sources. Conversely, the Falcon 40B model was trained on around 1 trillion tokens. These smaller models work better for those with computational and memory requirements or those who are worried that large models might overfit training data. The “Instruct” model is specifically fine-tuned to better follow human instructions, making it well-suited for interactive applications like chatbots.

Additionally, TII has released the Falcon 2 series in the following parameter sizes:

• Falcon 2 11B with 11 billion parameters
• Falcon 2 11B VLM (Vision-to-Language) with 11 billion parameters

The Falcon 2 11B model is a more efficient and accessible version compared to previous iterations and is trained on 5.5 trillion tokens. In fact, TII has stated that Falcon 2 11B surpasses the performance of Meta’s Llama 3 8B and performs on par with Google’s Gemma 7B. Falcon 2 models also have multilingual capabilities in English, French, Spanish, German, and more.

Falcon 2 11B VLM is notable in that it is TII’s first multimodal model and can convert visual inputs into text. Many LLMs have struggled with multimodal capabilities, and the Falcon 2 line is part of a new generation of LLMs to tackle this problem. What’s more, both Falcon 2 models run efficiently on a single GPU.

In the near future, Falcon 2 models will receive improvements like "Mixture of Experts” – a sophisticated machine learning feature. By combining smaller networks with discrete specializations, this approach makes sure that the most competent areas work together to provide complex and tailored solutions. It’s like having a group of knowledgeable assistants that collaborate to forecast or make judgments as necessary. Each assistant has a unique area of expertise.

Finally, one of the larger changes to the Falcon 2 series is the open-source approach. Original Falcon models came with some licensing restrictions. However, Falcon 2 models are released under a permissive open-source license, which gives developers worldwide unrestricted access to the tool.

Apple's 2024 Worldwide Developer Conference (WWDC) is already shaping up to be one of the company's biggest events in decades. The opening keynote, which is taking place right now, is focused almost entirely on the buzzword we can't stop talking about — artificial intelligence.

Trailing behind major players like OpenAI, Google, and Microsoft, Apple today is unveiling a slew of AI features spread across the company's most popular operating systems. While AI is the event's main focus, Apple executives are also expected to announce this year's software upgrades for the iPhone, iPad, Apple Watch, Mac, and Vision Pro.

If you can't tune in for the two-hour-long event, ZDNET has you covered. Here's a complete breakdown of all the announcements at WWDC as they come.

VisionOS 2

• Apple unveiled the first major upgrade to its recently released VisionOS: VisionOS 2.

• In Vision OS 2, Photos gets an upgrade that allows users to create Spatial Photos with added depth from photos already in their camera rolls.

• Spatial Personas in the Photos app lets users view photos together, creating a more shared experience.

• VisionOS 2 also supports new hand motion commands, allowing users to access some settings more easily. For example, users can open their hands and tap to reach the home screen or turn their wrists to see the battery level.

• Users who mirror their MacOS to their Vision Pro will soon get new sizes, including ultrawide monitor view.

• The Vision Pro will also include train support for travel mode, making working during your commute easier.

• Developers will be able to create Spatial Apps with more ease due to new frameworks and APIs.

• Apple is partnering with Blackmagic to make it easier to make Immersive Videos.
  

iOS 18

• iPhone and iPad users will be able to customize their home screen further by placing apps wherever they'd like on the screen, as opposed to the usual fixed grid. App icon colors will also be customizable, allowing users to make apps any color they want or even match their home screen. Users can also change app icons to dark mode.

• After five years of remaining untouched, the Control Center received several upgrades, including the ability to customize its toggles, like for flashlight, screen recording, calculator, auto-rotate, screen mirroring, and more, by tapping, holding, and rearranging. The Control Center toggle will also feature different pages with completely customizable controls for users. Developers can also create controls for their apps.

• Apple also added privacy options, including the ability to lock an app, which requires users to authenticate with FaceID or passcode before accessing the app. Users can also hide an app, which makes it disappear from the home screen to a hidden part of their app library.

• Messages received several upgrades. Tapbacks, the feature that allows users to react to messages by holding them down, was upgraded to feature different colors and include emojis. Users can add text effects to specific phrases or words instead of the entire phrase. Lastly, users will be able to schedule messages.

• iPhone 14 and later models will have a new Messages via Satellite feature, which allows users to send messages via satellite when they don't have Wi-Fi or cellular service.

• The Mail app will automatically categorize emails, a feature that will be available later this year.

• The Wallet app now allows users to tap phones together to exchange Apple Cash without requiring them to exchange personal information like phone numbers.

• The Journal app will now show more statistics and insights, including how many entries you've had this year, how many days you journaled, and more.

• There is a new Game Mode for iPhone, meant to help gamers optimize their gaming experience. This includes minimizing background activity and using more responsive accessories, such as controllers.

• The Photos app got what Apple dubbed its "biggest redesign ever," featuring a cleaner design and an improved search.

• Apple reiterated that RCS support will be coming to the iPhone.

• The Calendar app can now pull from the Reminders app for a more seamless overview of your schedule.

AirPods

• AirPods Pro are getting voice isolation to enhance call quality in noisier environments.

• You can now nod or shake your head "yes" or "no" when responding to Siri.

• Apple is also releasing a Personalized Spatial Audio API for game developers to build around the AirPods' audio technology.

tvOS 18

• When you watch an Apple TV show or movie, the new Insights section on tvOS will include additional information such as actor names and music titles. You can then easily add those music titles to your Apple Music playlist.

• Apple added support for 21:9 formatting for viewing widescreen films.

• tvOS will also feature a redesigned Apple Fitness+ experience, with enhanced dialogue, improved subtitles, and new AppleTV screen savers.

WatchOS 11

• The new Training Mode allows users to get insights into how the intensity of their workouts is impacting their performance in the long run.

• The new Vitals app will give users a quick look at their most important health metrics. If something seems out of the ordinary, users will receive pings alerting them of the anomaly.

• Cycle tracking is getting upgraded to better suit pregnancy, showing gestational age, and more.

• Smart Stack is also getting more intelligent, now able to automatically add widgets when needed and more.

iPadOS 18

• The update will feature a redesigned tab bar and side bar, which users can customize to showcase their favorite apps and access the most important sections of an app. You can also long-tap the bar to move it around.

• Shareplay will allow users to remotely control someone else's iPad or iPhone and share drawings on their screens.

• In a long-awaited release, iPads will now have a calculator app for the first time, complete with the same interface as the one currently found on iPhones. Plus, you can use it with the Apple Pencil through a new Math Notes experience, which allows users to write expressions that the calculator app will solve for you once you type the equal sign.

• Handwriting in Notes also got an upgrade with Smart Script, which refines users' writing to make it more legible while keeping the authenticity of the user's handwriting style. The feature can even match copied and pasted text to the user's handwriting. Spell check is also still compatible with it.

• iPadOS 18 supports screen-sharing via SharePlay and the same Control Center customizations and emoji Tapbacks found in iOS 18.

MacOS 15/Sequoia

• Apple unveiled MacOS Sequoia, which will feature many of the new features that were added to iOS 18 and iPadOS 18.

• The new iPhone mirroring capability on Mac allows users to experience their phone almost entirely from their Mac. For example, iPhone notifications will now be available on Mac, allowing users to interact with them and open corresponding apps, though the iPhone itself will appear locked.

• Video meetings are also getting an upgrade, with new backgrounds and previews that allow you to see what you are about to share before sharing it.

• Apple launched its take on password management services with its own Passwords app.

• The AI summarization tool will live in Safari to help users process content like web pages and articles more efficiently. Safari will also assist users in discovering more helpful information about a page they are browsing when relevant, such as directions.

• Apple also launched a new Viewer experience, which does for video what Reader does for text.

Apple Intelligence

• Apple unveiled what it calls its new "personal intelligence" system under the name Apple Intelligence. The release puts generative models at the heart of the ecosystem of Apple devices.

• With Apple Intelligence, your iPhone can prioritize notifications to ensure you get notified only when it's crucial throughout your day.

• The release includes writing tools that leverage AI, including rewriting, proofreading, and summarizing text features available across mail, keynotes, third-party apps, and more.

• Users can now create personalized images in the photo library, including sketches, illustrations, and animations. This feature is available in Messages, Apps, Freeform, Keynote, and Pages.

• Apple Intelligence can tap into tools and carry out tasks on your behalf, such as "Show me all the photos," "Play the podcast," or "Pull the files that my coworker shared with me last week."

• Because it's grounded in your personal information and context, and can retrieve data from across your apps and reference the content on your screen, Apple Intelligence is positioned to be your personal assistant.

• Apple emphasized the safety and privacy precautions built into Apple Intelligence, particularly for on-device intelligence processing. The company touted the security of Apple's silicon, A17 Pro, and its M family of chips (M1, M2, M3, and M4).

• For tasks that are too large for on-device processing and need to be completed into the cloud, Apple unveiled Private Cloud Compute, which protects users' privacy by running on servers specially created using Apple Silicon. When users make requests, Apple Intelligence first tests on-device capability, but calls on Private Cloud Compute if the task requires more compute power. Apple reiterated that user data is never stored or sold to external parties.

• Siri finally got the AI makeover it deserves, first with a new look: when tapped, a light wraps around the edges of your screen. Siri can now better understand users, even if they stutter, due to more advanced natural language processing (NLP). It now has conversational context, remembering what you just said and using it to carry out the next task. Users can also type requests to Siri. Because it has in-depth product knowledge, Siri can answer questions about functionality on iPad, iPhone, and Mac. Siri will also have Apple Intelligence's on-screen awareness, allowing it to take action on what it is viewing. The voice assistant can also take actions across apps, including photo editing. With access to your personal context, Siri can understand and complete new commands, such as pulling your driver's license information from a photo and automatically inputting it into a form. The Siri updates are coming to iPad and Mac, too.

• Apple Intelligence also powers new features in Mail, including Rewrite, which offers users different versions of what they have already written. Suggestions are shown in-line, and Proofread edits forgrammar, word choice, and sentence structure. You can also use Summarize to convert your text into bullet points. Smart Reply identifies intelligent selections of an email and uses them to help craft a custom message. Browsing an inbox will also be easier with summaries populated at the top of emails. Apple Intelligence can even help prioritize your emails, placing what is most important at the top of your inbox.

• There is an all-new focus option: reduce interruptions. When in this setting, your phone will only show you what is most important based on your personal activity and context.

• Genmoji allows users to create AI-generated emojis based on what they type. You can also create a Genmoji based on a photo of a friend. Genmojis can be included in-line in Messages and even used for Tapbacks.

• Image Playground allows users to leverage AI on-device to create images from text prompts, which can be easily shared in iMessage and elsewhere. The feature is also available in Keynote, Pages, and Freeform, and as a stand-alone Image Playground app.

• Image Wand in the Notes app transforms a rough sketch into a polished image and is available directly in the tool palette. For example, you can circle a rough sketch in Notes and open Image Playground to transform your doodle into a fully-fledged image.

• Apple Intelligence will also upgrade the Photos app with a new clean-up tool that removes unwanted objects. Search in videos allows users to easily find specific snippets of content, and users can create Memories on-demand, using text to edit and organize photos into movies.

• In the Notes app, users can record and transcribe audio, which Apple Intelligence will generate a text summary of. This experience is also available in the Notes phone app.

• Apple Intelligence is free on iOS 18, iPadOS 18, and MacOS Sequoia, and will be available to try in English only this summer.

Partnership with OpenAI

• Apple also confirmed its partnership with OpenAI by integrating ChatGPT with Siri, which can send a request to ChatGPT for help with a user's permission. For example, if you ask Siri for assistance on a task it deems ChatGPT could answer better, Siri will suggest you use ChatGPT instead. ChatGPT's writing capabilities can also be leveraged within certain writing tasks.

• Users can access ChatGPT via this integration for free, and their data will not be logged by OpenAI. ChatGPT Plus users can connect their subscriptions to access more advanced features.

• The ChatGPT integration will be coming to iOS 18, iPadOS 18, and MacOS Sequoia later this year.

Featured