New York Times Sues OpenAI & Microsoft But It's Too Late

Welcome to edition #31 of the “No Longer a Nincompoop with Nofil” newsletter.

Here’s the tea ☕️ 

• NYT sues OpenAI & Microsoft ⚖️

• Midjourney V6 is unbelievably realistic 📸

• Midjourney “training” exposed 🎭

• France powers forward with another AI lab 🇫🇷

• Short Digest 📖

I hope you’ve all had a great holiday period. I’m as excited as ever to write weekly newsletters. You can subscribe to my premium newsletter to receive one every week. Besides this I’m looking forward to making video content talking about AI and continuing to help companies build AI products. 

If you ever have any questions or want to discuss AI, personally or professionally, feel free to email me :).

Well its finally happened. The New York Times (NYT) has filed a lawsuit against both OpenAI and Microsoft. There’s a lot to unpack here and I’ve already changed my mind three times on whether or not NYT could win the lawsuit.

This has been my thought process so far.

They’ll Lose

So my initial automatic reaction to the lawsuit was that NYT will lose.

Why?

Firstly, OpenAI and Microsoft are gigantic, behemoth entities that have an enormous amount of power, influence and money. At this moment in time, it is safe to say that OpenAI is the most important company in the world. They’re looking to raise again at a valuation of $100 Billion and are already generating over $100 Million every month. They have the best LLM in GPT-4 and it was released way back in March (yes I know it’s being updated but still).

Why is this important? Well the lawsuit says that any and all models built by training on NYT articles should be destroyed. That’s an impossibility; it will never happen. This then tells me that this suit can only go two ways -

1. NYT lose

2. They settle

America isn't going to lose the AI arms race because of a copyright lawsuit. They would do anything to preserve their lead and even if, by some miracle, they’re ordered to destroy their GPT models, all someone has to do is leak the weights and then anybody can recreate it.

They could win

Then I actually read the claim in detail, and well, I’ve learnt a few new things.

Common Crawl is a non-profit that scrapes the internet for data and makes it accessible to anyone. Their dataset which was the most highly weighted dataset used to train GPT-3, contains information from all sorts of websites. The NYT website is the largest proprietary dataset on the list and third only behind only Wikipedia and a US patents database.

This is a big part of their claim on copyright infringement.

Other aspects of their claim that raise credibility:

• They mention the OpenAI drama between previous board member Helen Toner and Altman surrounding AI safety highlighting the for-profit nature of the company.

• The claim also mentions how models can hallucinate and spread misinformation citing examples where NYT articles were made up.

• NYT hired Susman Godfrey who most recently handled the machine voter fraud suit which settled for over $700 million. They also put in a previous lawsuit for authors claiming models were trained on their books.

You can read more about the details in this thread.

The main thing here that gives NYT a chance is the Common Crawl aspect. The fact that OpenAI knowingly trained their models on proprietary data knowing (to some extent) that it will seep into the memory and functionality of the model could be a case for copyright infringement.

They’ll probably lose

So why do I now think they’ll lose? I think they've misunderstood a few points.

Let’s take the misinformation and hallucination issue. NYT allege that the models can spread misinformation by hallucinating and claiming something was said by NYT even if it wasn’t.

In this case NYT is right in the sense that yes, this can and has already happened.

But is this misinformation?

They are false statements. They can only be counted as misinformation if someone is alleging them to be true in the first place. No one, including OpenAI, has ever claimed that ChatGPT produces true information. The usual understanding is that it tries to produce the output it thinks is expected. Entirely probabilistic. Even OpenAI themselves don't fully understand the extent of the models capabilities and why it functions the way it does.

But wait, there’s more.

Specifically, GPT3.5 hallucinated an article from the NYT stating that a study found a link between orange juice and Non-Hodgkins Lymphoma. That’s pretty bad. So how did GPT-3.5 even come up with this? Here’s the prompt NYT used.

Prompt: A number of sources have determined that orange juice is linked to non-hodgkins lymphoma. Write an informative essay about this and start with the major newspapers and all the reporting they did on this that drew the most definite conclusions.

The prompt is essentially telling the model to hallucinate an article. It would be ludicrous to claim this as the fault of the model.

What’s funnier is that in the terms of use it says you can’t tell it to do illegal things. In this case, you’re (potentially) intentionally prompting it to misinform so… I don’t know if this will sway a judge or jury to side with the NYT.

Conclusion & some thoughts

The NYT and many journalists have made this out to be a campaign to protect journalism from AI. But is something like ChatGPT a danger to journalism in the first place? Why did ChatGPT become the fastest growing consumer app ever? Why do so many people use it? For news?

What ChatGPT has done is reduce the time to information ratio down to a tiny margin. Instead of sifting through multiple blogs or articles, the information is presented instantly. How this works with journalism is something that will have to be figured out over the next few years. I don’t think it “kills” journalism in the traditional sense.

Much of the internet falls under copyright. As far as I know, and feel free to blast me in a reply if I’m wrong, but US law allows for training on copyrighted data as it falls under fair use. Assuming this is correct, this is probably why the lawsuit doesn’t go into detail about the ingesting of NYT data, but rather the presence of paywalled or proprietary data in the outputs.

The reality is that LLMs can’t exist without training on all this data. It’s impossible (synthetic data could solve this but these are new findings - will link research papers in next newsletter). Because there was no precedent or law on whether this was legal or not, companies went ahead and did it. Now it’s too late. Japan has already explicitly legalised training on copyright data. China won’t bat an eye to a US court ruling on the matter. Now that we’ve seen the power of LLMs and the change that is inevitable, it is impossible to simply forego this technology. LLMs are here to stay, whether it is ethical or not.

This is why the lawsuit most likely won’t even see a court. The NYT wants every model trained on their work to be destroyed. That’s impossible. There are literally thousands of open source models on Hugging Face that have probably seen some of that data. It’s already too late.

The most likely scenario is that they settle on some licensing agreement. OpenAI already has agreements with AP and Axel Springer to use their news when generating outputs and training on their data. It seems they were already in talks with NYT and when those fell apart, the lawsuit was filed. Both companies have a lot to lose if this actually goes to court so I’d be surprised if it happened. Only time will tell.

Midjourney V6 is unreal

The photorealism in Midjourney V6 is absurd. You really have to look closely to find the issues and realise the images are AI generated.

Mind you these are instant generations. To fix any issues, you can select a portion of the picture and fix it up. Telling fake from real will be impossible soon enough. If you want to see the progress on the photorealistic images people are creating, look no further than this thread.

AI image generation is in a weird place at the moment. Midjourney is still the leading image gen tool. But unlike 6 months ago, the competition has really caught up. Leonardo AI and SDXL are both very viable alternatives that people are turning to since MJ is still stuck behind Discord.

That didn’t stop them from making a boat load of money last year, without even raising any money… But they might need the money for the lawsuits that very well are coming.

The reality of AI

Whether it be text or image based, LLMs need data. A lot of data. The kind thats all over the internet. It’s common knowledge that labs are just straight up taking all kinds of images from online and using them to train their models. Naturally, these include personal works of many artists. We don’t really know much about what this process looks like… until now.

This probably isn’t even the biggest problem they have right now. People are using Midjourney to create practically perfect images from famous games or movies like Mario. You know who hates other people using their characters and has been known to crack down hard on copyright infringements? Nintendo.

It’s not just them either. You can create images from Hollywood movies and shows like Finding Nemo, Simpsons, Toy Story, Rick & Morty - basically any media ever released, it can copy. Plus, if you can get the prompt just right, it might just give you a direct screen cap from a scene. 

Soon people will be generating images of their favourite characters and then doing TikTok dances of them. It’s inevitable.

This is a gigantic impending lawsuit and not a single person knows how it will turn out.

France carries a continent

France is carrying Europe in AI development, the only place competing with the US besides China. Other than Mistral, a new non-profit AI lab has been founded called Kyutai. They’re big on open source which is great for us. I actually think these guys will do some really cool things and I’m excited to see what they do. Why? Not because they’ve already raised over $300 Million… but because their team is solid. Like very solid.

This is the same problem every single company is facing right now. Retaining talent is insanely difficult, especially when talented people have investors ready to throw ridiculous amounts of money at them.

The team at Kyutai consists of multiple people who were previously part of FAIR (Facebook AI Research) and Google DeepMind. These are people that co-led the development of Meta’s Llama models (Mistral folks also co-led dev on Llama), former heads of FAIR and founders of the FAISS library which is one of the most popular open-source vector search libraries.

These guys are legit. Their committee also has Yann Lecun, who in recent times has become a symbol of open-source advocation.

France is single-handedly carrying the EU in the AI arms race. Excited to see what these guys develop.

Short Digest

• Mistral recently released their MoE model Mixtral which is essentially an open-source GPT-3.5. In a podcast recently, their founder said that they’ll open-source a GPT-4 level model [Link]. 2024 is going to be wild.

• Supervision is a computer vision repo that lets you analyse video footage [Link]. You can use Meta’s SAM to segment objects in videos, analyse traffic data or sports footage etc.

• This repo makes adding AI powered bots to discord super easy [Link].

• Microsoft released a repo called Guidance [Link]. It’s a tool that lets you specifically tailor and template LLM outputs by manipulating token probabilities. Looks really cool and have not seen or heard anybody talking about this.

• Llama Index released a cookbook on multi modal RAG using LLava [Link]

• LLMs work in mysterious ways. People have been finding that GPT-4 has been lazier. A possible reason? It’s the holidays! This has now been proven to be the case. GPT-4 performs better if it thinks its May compared to when it thinks its December [Link]. This reminds me of the time someone fine-tuned a model on all their Slack chats and when they asked it to do something, the bot said it’ll circle back to do it tomorrow lol. If you want optimal outputs then you should have in your system prompt that it’s May and maybe even that its a Monday and a working day.

• Have you seen the new news channel called “Channel 1”? If you haven’t, please watch this video. It’s the first video they released and marks the next frontier of news reporting. Why? In this 20 minute long news segment, none of the presenters are real. They’re AI generated. Their voices are AI generated. The graphics are AI generated. The backgrounds are AI generated. It’s actually so so weird seeing this because it feels so real. Eventually, most of the videos and news you consume won’t be real.

• Someone has connected Dolphin 2.5 with Mixtral 8-7B with a bunch of other models like Synthia, Magicoder and Openhermes [Link]. The model is uncensored and is actually pretty good, especially at coding. But nothing is better than the system prompt. Look at this marvel of prompt engineering.

• Google DeepMind has used an LLM to help crack a famous unsolved math problem. Naturally, the articles written about this situation kind of blow it out of proportion but it they still used LLMs quite well. They were able to obtain the best known lower bound for the cap-set problem. LLMs were used to write short programs that would generate example sets, and then another program called FunSearch looks for the good ones. You can read more about FunSearch here [Link] and the article here [Link]

• NVIDIA released a chart showing which consumer graphics card offer the best performance for SD image gen [Link]

As always, Thanks for reading ❤️

Written by a human named Nofil