Apple Intelligence: Powered by OpenAI, Funded by Microsoft

Welcome to edition #21 of the NLAN Premium Newsletter.

Here’s the tea 🍵

• Apple Intelligence 🍎 

  • Their own AI 🤖 

  • Future of apps 📱 

  • Curious design choices 🖊️ 

  • Apple values security 🔒️ 

  • OpenAI’s involvement 🪟 

• China’s advance 🇨🇳 

  • Video models are getting good 📹️ 

Well it finally happened. Apple has added AI into iOS and, in very Apple fashion, called it Apple Intelligence.

I’m not going to explore all of the details on what exactly they’ve added, but I do want to discuss a few things.

Before I even begin, can we just acknowledge that OpenAI now has a partnership with both Microsoft and Apple?

Like, that is actually an impressive feat of deal making. Irrespective of what you think of OpenAI and the direction they’re headed, what they’ve managed to accomplish in the last 1.5 years is pretty astounding.

Now lets talk about Apple.

Not what you might think

Now I just want to make this clear, Apple isn’t going to send every single AI request you have to OpenAI. In fact, most people won’t even be able to use their AI features because they’ll only work for iPhone 15 models and up. Besides that, Apple has shown that the future of AI is smaller, on-device models.

That’s exactly what’s powering Apple’s AI on iPhones. They built their own custom ~3B size model and, according to benchmarks they’ve released, it’s actually quite good.

Mind you, these results aren’t comparing general knowledge between GPT-4 and Apple’s tiny 3B model. The first one is comparing instruction following, i.e “reply with 5 bullet points, containing info on X topics, limited to 500 words”. The second one is intuitive.

Instruction following is probably one of the most important things Apple needs to get right, and they’re on the right track. But why is it so important?

With their new “AI Infrastructure”, Apple can read every piece of data on your phone. For apps that opt-in, Siri will have access to all of that apps’ data, and will use it to create your “personal context”.

There is a catch though.

For Siri to use an app, developers will need to use Apple’s entirely new Intents framework. This framework will act as a kind of API to an app, and will call the app through Siri shortcuts. If developers don’t bother building with the Intents API, Siri won’t be able to use their app. Learn more about app intents here.

This begs the question. Why bother doing this as a developer?

If you build an app and people mostly use it through Siri, is that a good thing or bad thing?

Are they really using your app if they don’t even open it?

Actually, this is exactly where we are headed. This is the first sign that apps really might stop being used.

Let me explain.

The evolution of applications

Let’s assume that all developers build their apps so that Siri can access and use them because that’s what people want.

Now we have a situation where Siri literally has access to just about every single thing about you.

Siri:

• Knows what you’re looking at

• Can read the content on your screen

• Can access any control on your phone

• Can understand multi-turn instructions

• Has access to a shortcut system that can trigger all of your apps

• Understands just about every single thing about you

The perfect personal companion.

Apple will build a layer of abstraction over every single app so that Siri can use it. So now, when an app is released, you more so can use every single one of its features without even opening the app.

You know what comes next?

Developers will stop publishing apps with interfaces all together. They’ll just publish the API and docs so that Apple’s AI can understand and use it. Then, Siri will just act as the app interface for the end user.

I can definitely see this happening, and it’s not even far off. In fact, with these new updates, you could probably do this right now.

You know that demo OpenAI did last year about booking an Uber and a hotel?

We’re almost there.

So what does this mean for designers and UI engineers?

Bad things I imagine.

Naturally, not every single app is made to be used via voice and apps won’t just disappear, but change is certainly coming.

Side Note:
When I wrote the above paragraph, it made sense to me that not all apps would be voice controlled, and therefore, not all apps could be used via Siri.

I’ve come back to finish off this newsletter and have a different perspective.

Apple can simply change the front end for Siri.

Siri has the power to be the “Do-All” app. Do anything in any app via Siri, be it voice controlled, or some other front end Apple will design later on.

This is absolutely possible, and Apple isn’t alone in trying to get there.

The browser is dead, long live the browser

Subscribe to Premium to read the rest.