This incredible open-source model

Many AI algorithms struggle to answer over 1,000 words.

Chinese researchers improved models to overcome this issue.

Over the past year, AI chatbots have advanced greatly. In lengthier, more sophisticated interactions, ChatGPT, Claude, or Gemini can keep context. They can handle larger text inputs. While some programs can answer questions, they struggle with outputs beyond 1,000 words.

Chatbots cannot generate a credible story of thousands of words in a single response due to this reality. Many users have debated this limitation for months, questioning why. Tsinghua University researchers in Beijing believe they have identified the answer and have published their data for analysis.

A research team lead by Yushi Bai presented two large language models (LLMs) based on existing models that can generate “coherent” output of over 10,000 words. ‘The Metamorphosis’ by Franz Kafka is 11,000 words. LongWriter may write a medium-length book in one response. That’s impressive.

LLM output restriction is closely related to text length employed during supervised fine-tuning, say experts. They created AgentWrite to decompose generation chores and enable LLM to generate extended coherent outputs. Our work shows that the long-context LLM can handle a bigger output frame.

Using an open-source model from their university, GLM-4-9B, trained with 10 trillion tokens in 26 languages, the researchers created LongWriter-glm4-9b. They created LongWriter-llama3.1-8b using open-source Llama 3.1 model 8B. Both ideas can provide “coherent” replies above 10,000 words, according to their GitHub page.

The researchers asked a GLM-4 9B chatbot to write a 5,000-word tale, as shown in the video. The result? Maximum 1,896 words. The identical request is answered by LongWriter-glm4-9b with 7,872. Even with a limit, the new model seems to generate extended answers.

The good news is that anyone may examine and apply these developments. The open attitude many support in AI development has this benefit. A Chinese university can help a global project. However, corporations that construct closed models control all their advances.

Innovation is the future of the Chinese economy, and everyone knows it. That wasn’t always so. Over the past 50 years, China’s manufacturing miracle has lifted 700 million people out of poverty, but innovation didn’t drive it. The driver has mostly been brute-force imitation. China produced a lot of other countries’ breakthroughs, relying on a seemingly endless supply of cheap labor from the hundreds of millions of eager postwar baby boomers. The effort helped a country that missed the Industrial Revolution adopt the world’s most advanced industrial technologies in a decade or two. China became known as a global copycat.

Times are changing. New Millennials are replacing Baby Boomers in China, which introduced its one-child policy in 1979 to lower birth rates. It worked, but China now lacks enough Millennials and Gen Z workers to replace its aging Baby Boomers. National Bureau of Statistics: China will have 81 million fewer working-age adults in 2030 than in 2015; population will fall by 7.6 million annually after 2030. It has major consequences. China can no longer rely on imitation to expand and maintain its older population as its younger workforce shrinks. Instead, it must innovate.

But can China innovate? Can it compete globally with sophisticated nations that have built their economies on innovation for decades? Many are sceptical. They say the West has generated many innovations and innovators in recent years, whereas China has produced few. Our March 2014 piece, “Why China Can’t Innovate,” by Regina M. Abrami, William C. Kirby, and F. Warren McFarlan, caught conventional thinking. At the time, the authors’ arguments were solid. But two years later, eight of the 10 companies that attained a $1 billion valuation in the shortest period ever were Chinese, and six were launched the year the article was published.

These results are surprising for a country that ranked 14th on the Global Innovation Index in 2020. Some factor catapulted those Chinese companies to the top, but our innovation measures overlook it. We focus on personalities and enterprises with bold, creative ideas. That distinguishes the U.S. innovation ecosystem. China has reached parity with the US in the past five years as a “innovation cold war” has erupted, and its inventors may not be its main drivers.

This incredible open-source model

Many AI algorithms struggle to answer over 1,000 words.

By b0oua

Related Post

‘Alien Romulus’ is a powerful sequel that rivals

Star Wars: The Rise of Skywalker’ review

Apple’s $700 million investment,

Leave a Reply Cancel reply

You Missed

‘Alien Romulus’ is a powerful sequel that rivals

Star Wars: The Rise of Skywalker’ review

Apple’s $700 million investment,

How one of the great science fiction authors helped