Artificial General Intelligence

הערות · 216 צפיות

Artificial general intelligence (AGI) is a type of synthetic intelligence (AI) that matches or surpasses human cognitive capabilities across a large range of cognitive tasks.

Artificial basic intelligence (AGI) is a type of synthetic intelligence (AI) that matches or exceeds human cognitive capabilities throughout a wide variety of cognitive tasks. This contrasts with narrow AI, which is restricted to specific tasks. [1] Artificial superintelligence (ASI), on the other hand, describes AGI that greatly goes beyond human cognitive capabilities. AGI is considered among the definitions of strong AI.


Creating AGI is a main goal of AI research and of companies such as OpenAI [2] and Meta. [3] A 2020 survey identified 72 active AGI research study and development projects throughout 37 countries. [4]

The timeline for attaining AGI remains a subject of ongoing argument amongst scientists and professionals. As of 2023, some argue that it might be possible in years or years; others preserve it may take a century or longer; a minority think it may never ever be achieved; and another minority declares that it is currently here. [5] [6] Notable AI researcher Geoffrey Hinton has expressed concerns about the rapid development towards AGI, suggesting it could be accomplished sooner than lots of expect. [7]

There is dispute on the specific meaning of AGI and concerning whether contemporary big language designs (LLMs) such as GPT-4 are early kinds of AGI. [8] AGI is a typical subject in science fiction and futures research studies. [9] [10]

Contention exists over whether AGI represents an existential threat. [11] [12] [13] Many specialists on AI have actually stated that mitigating the risk of human termination positioned by AGI must be a global top priority. [14] [15] Others find the advancement of AGI to be too remote to provide such a threat. [16] [17]

Terminology


AGI is likewise referred to as strong AI, [18] [19] complete AI, [20] human-level AI, [5] human-level smart AI, or general intelligent action. [21]

Some scholastic sources schedule the term "strong AI" for computer system programs that experience life or awareness. [a] On the other hand, weak AI (or narrow AI) has the ability to fix one specific issue but lacks basic cognitive abilities. [22] [19] Some scholastic sources utilize "weak AI" to refer more broadly to any programs that neither experience consciousness nor have a mind in the same sense as people. [a]

Related principles consist of artificial superintelligence and transformative AI. An artificial superintelligence (ASI) is a theoretical kind of AGI that is much more generally smart than people, [23] while the concept of transformative AI relates to AI having a large effect on society, for instance, comparable to the agricultural or industrial transformation. [24]

A framework for classifying AGI in levels was proposed in 2023 by Google DeepMind scientists. They specify five levels of AGI: emerging, proficient, expert, virtuoso, and superhuman. For example, a competent AGI is specified as an AI that outperforms 50% of competent adults in a broad variety of non-physical jobs, and a superhuman AGI (i.e. a synthetic superintelligence) is likewise defined but with a limit of 100%. They consider large language designs like ChatGPT or LLaMA 2 to be instances of emerging AGI. [25]

Characteristics


Various popular definitions of intelligence have actually been proposed. One of the leading proposals is the Turing test. However, there are other widely known definitions, and some scientists disagree with the more popular techniques. [b]

Intelligence qualities


Researchers typically hold that intelligence is needed to do all of the following: [27]

factor, use method, fix puzzles, and make judgments under unpredictability
represent knowledge, consisting of good sense understanding
strategy
learn
- interact in natural language
- if needed, incorporate these abilities in completion of any provided objective


Many interdisciplinary techniques (e.g. cognitive science, computational intelligence, and choice making) think about additional qualities such as imagination (the capability to form novel psychological images and ideas) [28] and autonomy. [29]

Computer-based systems that show a lot of these abilities exist (e.g. see computational creativity, automated thinking, decision assistance system, robot, evolutionary calculation, smart agent). There is dispute about whether modern-day AI systems possess them to an adequate degree.


Physical characteristics


Other abilities are thought about desirable in smart systems, as they might affect intelligence or help in its expression. These consist of: [30]

- the capability to sense (e.g. see, hear, etc), and
- the capability to act (e.g. move and manipulate objects, modification location to explore, and so on).


This includes the ability to discover and respond to hazard. [31]

Although the capability to sense (e.g. see, hear, and so on) and the capability to act (e.g. relocation and manipulate objects, modification location to check out, etc) can be preferable for some intelligent systems, [30] these physical capabilities are not strictly needed for an entity to certify as AGI-particularly under the thesis that large language designs (LLMs) may currently be or become AGI. Even from a less positive point of view on LLMs, there is no firm requirement for an AGI to have a human-like type; being a silicon-based computational system is sufficient, offered it can process input (language) from the external world in location of human senses. This analysis aligns with the understanding that AGI has never ever been proscribed a particular physical personification and therefore does not demand a capacity for mobility or conventional "eyes and ears". [32]

Tests for human-level AGI


Several tests indicated to validate human-level AGI have been thought about, including: [33] [34]

The concept of the test is that the machine needs to try and pretend to be a man, by responding to concerns put to it, and it will only pass if the pretence is fairly convincing. A considerable portion of a jury, who should not be skilled about makers, should be taken in by the pretence. [37]

AI-complete issues


An issue is informally called "AI-complete" or "AI-hard" if it is believed that in order to solve it, one would need to implement AGI, because the option is beyond the abilities of a purpose-specific algorithm. [47]

There are lots of issues that have been conjectured to need basic intelligence to fix in addition to humans. Examples consist of computer vision, natural language understanding, and handling unforeseen situations while resolving any real-world issue. [48] Even a particular task like translation requires a maker to read and write in both languages, follow the author's argument (reason), understand the context (understanding), and consistently replicate the author's original intent (social intelligence). All of these issues need to be resolved concurrently in order to reach human-level maker performance.


However, a lot of these tasks can now be carried out by contemporary big language designs. According to Stanford University's 2024 AI index, AI has actually reached human-level performance on lots of benchmarks for reading comprehension and visual thinking. [49]

History


Classical AI


Modern AI research study started in the mid-1950s. [50] The first generation of AI researchers were encouraged that synthetic basic intelligence was possible and that it would exist in just a few decades. [51] AI leader Herbert A. Simon wrote in 1965: "machines will be capable, within twenty years, of doing any work a male can do." [52]

Their forecasts were the inspiration for Stanley Kubrick and Arthur C. Clarke's character HAL 9000, who embodied what AI scientists believed they might create by the year 2001. AI leader Marvin Minsky was an expert [53] on the job of making HAL 9000 as practical as possible according to the agreement forecasts of the time. He said in 1967, "Within a generation ... the problem of producing 'expert system' will significantly be fixed". [54]

Several classical AI jobs, such as Doug Lenat's Cyc job (that began in 1984), and Allen Newell's Soar job, were directed at AGI.


However, in the early 1970s, it ended up being obvious that researchers had actually grossly underestimated the difficulty of the project. Funding firms ended up being skeptical of AGI and put scientists under increasing pressure to produce useful "used AI". [c] In the early 1980s, Japan's Fifth Generation Computer Project restored interest in AGI, setting out a ten-year timeline that included AGI objectives like "bring on a casual conversation". [58] In reaction to this and the success of expert systems, both industry and government pumped money into the field. [56] [59] However, confidence in AI spectacularly collapsed in the late 1980s, and the goals of the Fifth Generation Computer Project were never fulfilled. [60] For the second time in 20 years, AI scientists who anticipated the impending accomplishment of AGI had actually been misinterpreted. By the 1990s, AI scientists had a credibility for making vain promises. They became hesitant to make forecasts at all [d] and prevented mention of "human level" synthetic intelligence for worry of being labeled "wild-eyed dreamer [s]. [62]

Narrow AI research


In the 1990s and early 21st century, mainstream AI accomplished commercial success and academic respectability by concentrating on specific sub-problems where AI can produce verifiable results and commercial applications, such as speech acknowledgment and recommendation algorithms. [63] These "applied AI" systems are now used thoroughly throughout the innovation market, and research study in this vein is heavily moneyed in both academia and industry. As of 2018 [update], advancement in this field was considered an emerging trend, and a fully grown stage was anticipated to be reached in more than 10 years. [64]

At the turn of the century, lots of mainstream AI researchers [65] hoped that strong AI might be developed by integrating programs that fix different sub-problems. Hans Moravec composed in 1988:


I am confident that this bottom-up route to synthetic intelligence will one day satisfy the traditional top-down path majority method, prepared to offer the real-world proficiency and the commonsense knowledge that has been so frustratingly evasive in thinking programs. Fully smart machines will result when the metaphorical golden spike is driven uniting the two efforts. [65]

However, even at the time, this was disputed. For instance, Stevan Harnad of Princeton University concluded his 1990 paper on the sign grounding hypothesis by stating:


The expectation has typically been voiced that "top-down" (symbolic) approaches to modeling cognition will somehow satisfy "bottom-up" (sensory) approaches someplace in between. If the grounding factors to consider in this paper stand, then this expectation is hopelessly modular and there is truly just one feasible path from sense to signs: from the ground up. A free-floating symbolic level like the software level of a computer system will never be reached by this route (or vice versa) - nor is it clear why we need to even attempt to reach such a level, since it appears arriving would just amount to uprooting our signs from their intrinsic significances (thereby simply minimizing ourselves to the practical equivalent of a programmable computer system). [66]

Modern synthetic general intelligence research study


The term "artificial basic intelligence" was used as early as 1997, by Mark Gubrud [67] in a discussion of the ramifications of fully automated military production and operations. A mathematical formalism of AGI was proposed by Marcus Hutter in 2000. Named AIXI, the proposed AGI representative maximises "the capability to please objectives in a vast array of environments". [68] This kind of AGI, defined by the capability to maximise a mathematical definition of intelligence rather than show human-like behaviour, [69] was likewise called universal synthetic intelligence. [70]

The term AGI was re-introduced and promoted by Shane Legg and Ben Goertzel around 2002. [71] AGI research activity in 2006 was described by Pei Wang and Ben Goertzel [72] as "producing publications and initial results". The first summer season school in AGI was arranged in Xiamen, China in 2009 [73] by the Xiamen university's Artificial Brain Laboratory and OpenCog. The very first university course was given up 2010 [74] and 2011 [75] at Plovdiv University, Bulgaria by Todor Arnaudov. MIT presented a course on AGI in 2018, organized by Lex Fridman and featuring a variety of guest speakers.


As of 2023 [upgrade], a little number of computer system scientists are active in AGI research, and lots of add to a series of AGI conferences. However, significantly more researchers are interested in open-ended learning, [76] [77] which is the idea of allowing AI to continually learn and innovate like human beings do.


Feasibility


As of 2023, the development and potential achievement of AGI remains a topic of intense dispute within the AI neighborhood. While conventional consensus held that AGI was a far-off objective, recent developments have led some researchers and industry figures to claim that early forms of AGI might already exist. [78] AI leader Herbert A. Simon hypothesized in 1965 that "makers will be capable, within twenty years, of doing any work a guy can do". This prediction stopped working to come real. Microsoft co-founder Paul Allen thought that such intelligence is unlikely in the 21st century because it would need "unforeseeable and fundamentally unpredictable developments" and a "scientifically deep understanding of cognition". [79] Writing in The Guardian, roboticist Alan Winfield claimed the gulf between modern-day computing and human-level expert system is as wide as the gulf in between existing area flight and practical faster-than-light spaceflight. [80]

A further difficulty is the absence of clearness in defining what intelligence involves. Does it require awareness? Must it display the ability to set goals in addition to pursue them? Is it purely a matter of scale such that if design sizes increase sufficiently, intelligence will emerge? Are facilities such as planning, reasoning, and causal understanding needed? Does intelligence need clearly replicating the brain and its specific faculties? Does it need feelings? [81]

Most AI researchers think strong AI can be achieved in the future, however some thinkers, like Hubert Dreyfus and Roger Penrose, deny the possibility of achieving strong AI. [82] [83] John McCarthy is among those who believe human-level AI will be accomplished, however that today level of development is such that a date can not accurately be predicted. [84] AI experts' views on the expediency of AGI wax and subside. Four surveys conducted in 2012 and 2013 suggested that the average price quote amongst professionals for when they would be 50% confident AGI would get here was 2040 to 2050, depending on the survey, with the mean being 2081. Of the professionals, 16.5% answered with "never" when asked the very same question however with a 90% confidence rather. [85] [86] Further existing AGI progress considerations can be discovered above Tests for confirming human-level AGI.


A report by Stuart Armstrong and Kaj Sotala of the Machine Intelligence Research Institute found that "over [a] 60-year time frame there is a strong bias towards predicting the arrival of human-level AI as between 15 and 25 years from the time the prediction was made". They analyzed 95 forecasts made between 1950 and 2012 on when human-level AI will come about. [87]

In 2023, Microsoft researchers released a comprehensive examination of GPT-4. They concluded: "Given the breadth and depth of GPT-4's abilities, our company believe that it could reasonably be considered as an early (yet still incomplete) variation of a synthetic general intelligence (AGI) system." [88] Another study in 2023 reported that GPT-4 exceeds 99% of people on the Torrance tests of creativity. [89] [90]

Blaise Agüera y Arcas and Peter Norvig composed in 2023 that a significant level of basic intelligence has already been accomplished with frontier designs. They wrote that reluctance to this view originates from 4 primary factors: a "healthy hesitation about metrics for AGI", an "ideological dedication to alternative AI theories or strategies", a "dedication to human (or biological) exceptionalism", or a "concern about the economic implications of AGI". [91]

2023 likewise marked the development of large multimodal models (big language designs efficient in processing or creating several methods such as text, audio, and images). [92]

In 2024, OpenAI launched o1-preview, the first of a series of models that "spend more time believing before they respond". According to Mira Murati, this capability to think before reacting represents a brand-new, extra paradigm. It improves design outputs by spending more computing power when producing the answer, whereas the design scaling paradigm improves outputs by increasing the model size, training information and training compute power. [93] [94]

An OpenAI employee, Vahid Kazemi, claimed in 2024 that the company had accomplished AGI, specifying, "In my opinion, we have actually already attained AGI and it's even more clear with O1." Kazemi clarified that while the AI is not yet "better than any human at any task", it is "much better than a lot of humans at the majority of jobs." He also dealt with criticisms that big language designs (LLMs) merely follow predefined patterns, comparing their learning process to the clinical technique of observing, assuming, and verifying. These statements have actually stimulated argument, as they rely on a broad and unconventional definition of AGI-traditionally understood as AI that matches human intelligence throughout all domains. Critics argue that, while OpenAI's designs demonstrate amazing flexibility, they might not completely meet this requirement. Notably, Kazemi's comments came shortly after OpenAI got rid of "AGI" from the terms of its collaboration with Microsoft, prompting speculation about the company's tactical objectives. [95]

Timescales


Progress in expert system has historically gone through durations of quick development separated by durations when development appeared to stop. [82] Ending each hiatus were fundamental advances in hardware, software application or both to create space for additional development. [82] [98] [99] For example, the computer hardware readily available in the twentieth century was not sufficient to execute deep knowing, which needs great deals of GPU-enabled CPUs. [100]

In the introduction to his 2006 book, [101] Goertzel says that estimates of the time required before a genuinely versatile AGI is developed differ from ten years to over a century. Since 2007 [upgrade], the consensus in the AGI research study neighborhood seemed to be that the timeline talked about by Ray Kurzweil in 2005 in The Singularity is Near [102] (i.e. in between 2015 and 2045) was plausible. [103] Mainstream AI researchers have actually offered a vast array of opinions on whether progress will be this fast. A 2012 meta-analysis of 95 such opinions discovered a bias towards anticipating that the start of AGI would occur within 16-26 years for modern-day and historic predictions alike. That paper has been criticized for how it classified opinions as specialist or non-expert. [104]

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton developed a neural network called AlexNet, which won the ImageNet competitors with a top-5 test mistake rate of 15.3%, significantly much better than the second-best entry's rate of 26.3% (the conventional technique utilized a weighted sum of scores from various pre-defined classifiers). [105] AlexNet was considered the initial ground-breaker of the present deep knowing wave. [105]

In 2017, researchers Feng Liu, Yong Shi, and Ying Liu carried out intelligence tests on publicly readily available and easily available weak AI such as Google AI, Apple's Siri, and others. At the maximum, these AIs reached an IQ worth of about 47, which corresponds around to a six-year-old child in very first grade. An adult comes to about 100 usually. Similar tests were performed in 2014, with the IQ rating reaching a maximum worth of 27. [106] [107]

In 2020, OpenAI developed GPT-3, a language design capable of performing many varied tasks without specific training. According to Gary Grossman in a VentureBeat article, while there is agreement that GPT-3 is not an example of AGI, it is considered by some to be too advanced to be categorized as a narrow AI system. [108]

In the very same year, Jason Rohrer used his GPT-3 account to develop a chatbot, and offered a chatbot-developing platform called "Project December". OpenAI asked for modifications to the chatbot to comply with their security standards; Rohrer detached Project December from the GPT-3 API. [109]

In 2022, DeepMind established Gato, a "general-purpose" system capable of carrying out more than 600 various jobs. [110]

In 2023, Microsoft Research released a research study on an early variation of OpenAI's GPT-4, contending that it displayed more basic intelligence than previous AI designs and showed human-level performance in tasks spanning several domains, such as mathematics, coding, and law. This research study stimulated an argument on whether GPT-4 might be thought about an early, insufficient variation of synthetic general intelligence, stressing the requirement for further expedition and assessment of such systems. [111]

In 2023, the AI scientist Geoffrey Hinton stated that: [112]

The idea that this things might really get smarter than individuals - a couple of individuals thought that, [...] But the majority of people believed it was way off. And I believed it was way off. I thought it was 30 to 50 years or even longer away. Obviously, I no longer believe that.


In May 2023, Demis Hassabis likewise said that "The progress in the last couple of years has been quite unbelievable", which he sees no reason that it would slow down, anticipating AGI within a decade or perhaps a couple of years. [113] In March 2024, Nvidia's CEO, Jensen Huang, stated his expectation that within five years, AI would can passing any test a minimum of as well as human beings. [114] In June 2024, the AI researcher Leopold Aschenbrenner, a former OpenAI employee, estimated AGI by 2027 to be "noticeably possible". [115]

Whole brain emulation


While the development of transformer models like in ChatGPT is thought about the most promising course to AGI, [116] [117] whole brain emulation can serve as an alternative method. With whole brain simulation, a brain model is developed by scanning and mapping a biological brain in information, and then copying and simulating it on a computer system or another computational device. The simulation design must be sufficiently devoted to the initial, so that it behaves in practically the very same way as the initial brain. [118] Whole brain emulation is a type of brain simulation that is discussed in computational neuroscience and neuroinformatics, and for medical research purposes. It has been gone over in artificial intelligence research [103] as a method to strong AI. Neuroimaging technologies that might deliver the essential in-depth understanding are improving rapidly, and futurist Ray Kurzweil in the book The Singularity Is Near [102] forecasts that a map of enough quality will appear on a comparable timescale to the computing power needed to emulate it.


Early approximates


For low-level brain simulation, a very effective cluster of computer systems or GPUs would be required, offered the enormous amount of synapses within the human brain. Each of the 1011 (one hundred billion) nerve cells has on typical 7,000 synaptic connections (synapses) to other nerve cells. The brain of a three-year-old child has about 1015 synapses (1 quadrillion). This number decreases with age, stabilizing by adulthood. Estimates vary for an adult, varying from 1014 to 5 × 1014 synapses (100 to 500 trillion). [120] A quote of the brain's processing power, based upon an easy switch design for neuron activity, is around 1014 (100 trillion) synaptic updates per second (SUPS). [121]

In 1997, Kurzweil took a look at numerous estimates for the hardware needed to equate to the human brain and embraced a figure of 1016 computations per 2nd (cps). [e] (For contrast, if a "calculation" was equivalent to one "floating-point operation" - a step utilized to rate present supercomputers - then 1016 "computations" would be comparable to 10 petaFLOPS, accomplished in 2011, while 1018 was accomplished in 2022.) He utilized this figure to anticipate the essential hardware would be offered at some point between 2015 and 2025, if the rapid development in computer system power at the time of composing continued.


Current research study


The Human Brain Project, an EU-funded effort active from 2013 to 2023, has actually developed a particularly in-depth and publicly available atlas of the human brain. [124] In 2023, scientists from Duke University performed a high-resolution scan of a mouse brain.


Criticisms of simulation-based methods


The synthetic neuron design assumed by Kurzweil and utilized in lots of current artificial neural network implementations is basic compared with biological neurons. A brain simulation would likely need to catch the comprehensive cellular behaviour of biological nerve cells, currently understood only in broad outline. The overhead introduced by full modeling of the biological, chemical, and physical information of neural behaviour (particularly on a molecular scale) would require computational powers numerous orders of magnitude larger than Kurzweil's quote. In addition, the quotes do not represent glial cells, which are understood to contribute in cognitive procedures. [125]

A basic criticism of the simulated brain method originates from embodied cognition theory which asserts that human personification is a vital aspect of human intelligence and is necessary to ground meaning. [126] [127] If this theory is right, any fully practical brain model will require to incorporate more than just the neurons (e.g., a robotic body). Goertzel [103] proposes virtual personification (like in metaverses like Second Life) as an option, however it is unidentified whether this would suffice.


Philosophical point of view


"Strong AI" as specified in philosophy


In 1980, thinker John Searle coined the term "strong AI" as part of his Chinese space argument. [128] He proposed a difference between two hypotheses about artificial intelligence: [f]

Strong AI hypothesis: An artificial intelligence system can have "a mind" and "awareness".
Weak AI hypothesis: An expert system system can (just) imitate it thinks and has a mind and consciousness.


The first one he called "strong" due to the fact that it makes a more powerful statement: it assumes something unique has occurred to the maker that goes beyond those abilities that we can evaluate. The behaviour of a "weak AI" machine would be exactly identical to a "strong AI" device, however the latter would likewise have subjective conscious experience. This usage is also common in academic AI research study and books. [129]

In contrast to Searle and mainstream AI, some futurists such as Ray Kurzweil use the term "strong AI" to suggest "human level synthetic basic intelligence". [102] This is not the same as Searle's strong AI, unless it is presumed that consciousness is required for human-level AGI. Academic thinkers such as Searle do not think that holds true, and to most synthetic intelligence researchers the concern is out-of-scope. [130]

Mainstream AI is most thinking about how a program acts. [131] According to Russell and Norvig, "as long as the program works, they do not care if you call it real or a simulation." [130] If the program can behave as if it has a mind, then there is no need to know if it really has mind - indeed, there would be no chance to tell. For AI research study, Searle's "weak AI hypothesis" is equivalent to the declaration "synthetic general intelligence is possible". Thus, according to Russell and Norvig, "most AI researchers take the weak AI hypothesis for granted, and don't care about the strong AI hypothesis." [130] Thus, for scholastic AI research, "Strong AI" and "AGI" are two different things.


Consciousness


Consciousness can have different meanings, and some elements play substantial functions in sci-fi and the principles of expert system:


Sentience (or "remarkable consciousness"): The ability to "feel" understandings or feelings subjectively, rather than the ability to reason about perceptions. Some thinkers, such as David Chalmers, use the term "awareness" to refer exclusively to phenomenal awareness, which is approximately equivalent to sentience. [132] Determining why and how subjective experience arises is referred to as the tough problem of awareness. [133] Thomas Nagel discussed in 1974 that it "feels like" something to be mindful. If we are not conscious, then it does not seem like anything. Nagel utilizes the example of a bat: we can smartly ask "what does it feel like to be a bat?" However, we are not likely to ask "what does it feel like to be a toaster?" Nagel concludes that a bat seems conscious (i.e., has awareness) but a toaster does not. [134] In 2022, a Google engineer claimed that the company's AI chatbot, LaMDA, had actually attained life, though this claim was widely challenged by other experts. [135]

Self-awareness: To have conscious awareness of oneself as a different person, particularly to be consciously conscious of one's own thoughts. This is opposed to merely being the "topic of one's thought"-an operating system or debugger has the ability to be "knowledgeable about itself" (that is, to represent itself in the very same way it represents everything else)-however this is not what people typically suggest when they use the term "self-awareness". [g]

These characteristics have a moral dimension. AI life would generate issues of well-being and legal defense, likewise to animals. [136] Other elements of awareness associated to cognitive abilities are also relevant to the principle of AI rights. [137] Figuring out how to incorporate innovative AI with existing legal and social frameworks is an emergent problem. [138]

Benefits


AGI could have a large range of applications. If oriented towards such objectives, AGI might help mitigate various issues worldwide such as hunger, hardship and illness. [139]

AGI could improve performance and effectiveness in a lot of tasks. For instance, in public health, AGI could speed up medical research study, especially against cancer. [140] It might look after the elderly, [141] and equalize access to quick, top quality medical diagnostics. It could use enjoyable, low-cost and tailored education. [141] The requirement to work to subsist could become outdated if the wealth produced is effectively rearranged. [141] [142] This likewise raises the question of the location of human beings in a significantly automated society.


AGI could likewise assist to make reasonable decisions, and to prepare for and avoid disasters. It could likewise help to reap the advantages of potentially devastating technologies such as nanotechnology or environment engineering, while avoiding the associated dangers. [143] If an AGI's primary objective is to avoid existential catastrophes such as human termination (which might be tough if the Vulnerable World Hypothesis ends up being true), [144] it could take procedures to drastically minimize the dangers [143] while lessening the effect of these measures on our lifestyle.


Risks


Existential dangers


AGI might represent multiple kinds of existential risk, which are risks that threaten "the early extinction of Earth-originating smart life or the long-term and extreme destruction of its capacity for preferable future development". [145] The risk of human extinction from AGI has been the subject of many arguments, however there is also the possibility that the development of AGI would lead to a permanently flawed future. Notably, it could be used to spread and preserve the set of values of whoever develops it. If mankind still has moral blind areas similar to slavery in the past, AGI might irreversibly entrench it, avoiding moral progress. [146] Furthermore, AGI could assist in mass monitoring and indoctrination, which could be utilized to produce a steady repressive around the world totalitarian routine. [147] [148] There is likewise a danger for the devices themselves. If makers that are sentient or otherwise worthwhile of moral consideration are mass produced in the future, taking part in a civilizational course that indefinitely ignores their well-being and interests could be an existential disaster. [149] [150] Considering how much AGI could improve mankind's future and aid minimize other existential dangers, Toby Ord calls these existential threats "an argument for continuing with due care", not for "abandoning AI". [147]

Risk of loss of control and human termination


The thesis that AI poses an existential danger for people, which this danger requires more attention, is questionable however has actually been endorsed in 2023 by lots of public figures, AI researchers and CEOs of AI companies such as Elon Musk, Bill Gates, Geoffrey Hinton, Yoshua Bengio, Demis Hassabis and Sam Altman. [151] [152]

In 2014, Stephen Hawking slammed extensive indifference:


So, facing possible futures of enormous benefits and dangers, the professionals are surely doing whatever possible to make sure the best outcome, right? Wrong. If a superior alien civilisation sent us a message stating, 'We'll get here in a couple of decades,' would we just reply, 'OK, call us when you get here-we'll leave the lights on?' Probably not-but this is more or less what is occurring with AI. [153]

The possible fate of humanity has often been compared to the fate of gorillas threatened by human activities. The contrast states that greater intelligence permitted humanity to dominate gorillas, which are now susceptible in ways that they could not have actually expected. As a result, the gorilla has become an endangered species, not out of malice, but merely as a security damage from human activities. [154]

The skeptic Yann LeCun considers that AGIs will have no desire to control mankind which we need to beware not to anthropomorphize them and analyze their intents as we would for human beings. He stated that people won't be "clever adequate to create super-intelligent devices, yet ridiculously foolish to the point of offering it moronic goals with no safeguards". [155] On the other side, the concept of important convergence suggests that practically whatever their goals, smart agents will have factors to try to make it through and get more power as intermediary steps to achieving these objectives. And that this does not require having emotions. [156]

Many scholars who are concerned about existential danger advocate for more research into solving the "control problem" to answer the question: what types of safeguards, algorithms, or architectures can programmers carry out to increase the likelihood that their recursively-improving AI would continue to behave in a friendly, instead of devastating, way after it reaches superintelligence? [157] [158] Solving the control issue is made complex by the AI arms race (which might result in a race to the bottom of security precautions in order to release items before competitors), [159] and using AI in weapon systems. [160]

The thesis that AI can pose existential danger also has critics. Skeptics typically say that AGI is not likely in the short-term, or that issues about AGI distract from other issues related to current AI. [161] Former Google fraud czar Shuman Ghosemajumder considers that for many individuals outside of the innovation market, existing chatbots and LLMs are currently viewed as though they were AGI, leading to further misconception and fear. [162]

Skeptics often charge that the thesis is crypto-religious, with an unreasonable belief in the possibility of superintelligence replacing an illogical belief in an omnipotent God. [163] Some scientists believe that the interaction campaigns on AI existential danger by certain AI groups (such as OpenAI, Anthropic, DeepMind, and Conjecture) might be an at effort at regulatory capture and to inflate interest in their products. [164] [165]

In 2023, the CEOs of Google DeepMind, OpenAI and Anthropic, together with other market leaders and scientists, provided a joint statement asserting that "Mitigating the danger of extinction from AI need to be an international top priority alongside other societal-scale dangers such as pandemics and surgiteams.com nuclear war." [152]

Mass unemployment


Researchers from OpenAI approximated that "80% of the U.S. workforce could have at least 10% of their work tasks impacted by the intro of LLMs, while around 19% of employees might see at least 50% of their tasks affected". [166] [167] They consider workplace workers to be the most exposed, for instance mathematicians, accounting professionals or web designers. [167] AGI could have a much better autonomy, capability to make choices, to user interface with other computer tools, however also to control robotized bodies.


According to Stephen Hawking, the outcome of automation on the lifestyle will depend on how the wealth will be redistributed: [142]

Everyone can enjoy a life of elegant leisure if the machine-produced wealth is shared, or the majority of people can wind up badly bad if the machine-owners successfully lobby against wealth redistribution. Up until now, the trend seems to be toward the 2nd choice, with innovation driving ever-increasing inequality


Elon Musk considers that the automation of society will require federal governments to adopt a universal standard income. [168]

See likewise


Artificial brain - Software and hardware with cognitive abilities similar to those of the animal or human brain
AI result
AI security - Research location on making AI safe and helpful
AI positioning - AI conformance to the intended objective
A.I. Rising - 2018 movie directed by Lazar Bodroža
Expert system
Automated machine learning - Process of automating the application of artificial intelligence
BRAIN Initiative - Collaborative public-private research study initiative revealed by the Obama administration
China Brain Project
Future of Humanity Institute - Defunct Oxford interdisciplinary research study centre
General video game playing - Ability of expert system to play various games
Generative expert system - AI system efficient in generating material in action to prompts
Human Brain Project - Scientific research job
Intelligence amplification - Use of details innovation to enhance human intelligence (IA).
Machine ethics - Moral behaviours of man-made devices.
Moravec's paradox.
Multi-task learning - Solving multiple maker discovering tasks at the very same time.
Neural scaling law - Statistical law in artificial intelligence.
Outline of synthetic intelligence - Overview of and topical guide to expert system.
Transhumanism - Philosophical movement.
Synthetic intelligence - Alternate term for or form of expert system.
Transfer learning - Artificial intelligence technique.
Loebner Prize - Annual AI competition.
Hardware for expert system - Hardware specially created and optimized for artificial intelligence.
Weak artificial intelligence - Form of artificial intelligence.


Notes


^ a b See listed below for the origin of the term "strong AI", and see the scholastic definition of "strong AI" and weak AI in the article Chinese room.
^ AI creator John McCarthy writes: "we can not yet define in basic what kinds of computational procedures we wish to call intelligent. " [26] (For a conversation of some meanings of intelligence used by expert system researchers, see approach of expert system.).
^ The Lighthill report specifically slammed AI's "grandiose goals" and led the dismantling of AI research in England. [55] In the U.S., DARPA ended up being figured out to fund just "mission-oriented direct research study, wiki.myamens.com instead of fundamental undirected research study". [56] [57] ^ As AI creator John McCarthy writes "it would be a great relief to the remainder of the employees in AI if the creators of brand-new basic formalisms would reveal their hopes in a more protected type than has actually sometimes been the case." [61] ^ In "Mind Children" [122] 1015 cps is used. More just recently, in 1997, [123] Moravec argued for 108 MIPS which would approximately represent 1014 cps. Moravec talks in terms of MIPS, not "cps", which is a non-standard term Kurzweil presented.
^ As defined in a standard AI book: "The assertion that machines might potentially act wisely (or, maybe much better, kenpoguy.com act as if they were smart) is called the 'weak AI' hypothesis by philosophers, and the assertion that machines that do so are actually thinking (instead of imitating thinking) is called the 'strong AI' hypothesis." [121] ^ Alan Turing made this point in 1950. [36] References


^ Krishna, Sri (9 February 2023). "What is artificial narrow intelligence (ANI)?". VentureBeat. Retrieved 1 March 2024. ANI is created to perform a single task.
^ "OpenAI Charter". OpenAI. Retrieved 6 April 2023. Our objective is to make sure that artificial basic intelligence advantages all of humanity.
^ Heath, Alex (18 January 2024). "Mark Zuckerberg's new goal is producing synthetic general intelligence". The Verge. Retrieved 13 June 2024. Our vision is to build AI that is better than human-level at all of the human senses.
^ Baum, Seth D. (2020 ). A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy (PDF) (Report). Global Catastrophic Risk Institute. Retrieved 28 November 2024. 72 AGI R&D tasks were recognized as being active in 2020.
^ a b c "AI timelines: What do specialists in expert system expect for the future?". Our World in Data. Retrieved 6 April 2023.
^ Metz, Cade (15 May 2023). "Some Researchers Say A.I. Is Already Here, Stirring Debate in Tech Circles". The New York City Times. Retrieved 18 May 2023.
^ "AI pioneer Geoffrey Hinton stops Google and cautions of danger ahead". The New York Times. 1 May 2023. Retrieved 2 May 2023. It is difficult to see how you can avoid the bad stars from using it for bad things.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric (2023 ). "Sparks of Artificial General Intelligence: Early try outs GPT-4". arXiv preprint. arXiv:2303.12712. GPT-4 reveals triggers of AGI.
^ Butler, Octavia E. (1993 ). Parable of the Sower. Grand Central Publishing. ISBN 978-0-4466-7550-5. All that you touch you change. All that you change modifications you.
^ Vinge, Vernor (1992 ). A Fire Upon the Deep. Tor Books. ISBN 978-0-8125-1528-2. The Singularity is coming.
^ Morozov, Evgeny (30 June 2023). "The True Threat of Expert System". The New York City Times. The genuine risk is not AI itself however the method we release it.
^ "Impressed by expert system? Experts state AGI is following, and it has 'existential' threats". ABC News. 23 March 2023. Retrieved 6 April 2023. AGI might position existential threats to humanity.
^ Bostrom, Nick (2014 ). Superintelligence: Paths, Dangers, Strategies. Oxford University Press. ISBN 978-0-1996-7811-2. The very first superintelligence will be the last innovation that humanity needs to make.
^ Roose, Kevin (30 May 2023). "A.I. Poses 'Risk of Extinction,' Industry Leaders Warn". The New York City Times. Mitigating the threat of termination from AI ought to be an international concern.
^ "Statement on AI Risk". Center for AI Safety. Retrieved 1 March 2024. AI experts warn of danger of termination from AI.
^ Mitchell, Melanie (30 May 2023). "Are AI's Doomsday Scenarios Worth Taking Seriously?". The New York Times. We are far from developing makers that can outthink us in general ways.
^ LeCun, Yann (June 2023). "AGI does not present an existential danger". Medium. There is no reason to fear AI as an existential hazard.
^ Kurzweil 2005, p. 260.
^ a b Kurzweil, Ray (5 August 2005), "Long Live AI", Forbes, archived from the initial on 14 August 2005: Kurzweil explains strong AI as "machine intelligence with the complete series of human intelligence.".
^ "The Age of Expert System: George John at TEDxLondonBusinessSchool 2013". Archived from the initial on 26 February 2014. Retrieved 22 February 2014.
^ Newell & Simon 1976, This is the term they utilize for "human-level" intelligence in the physical symbol system hypothesis.
^ "The Open University on Strong and Weak AI". Archived from the initial on 25 September 2009. Retrieved 8 October 2007.
^ "What is synthetic superintelligence (ASI)?|Definition from TechTarget". Enterprise AI. Retrieved 8 October 2023.
^ "Artificial intelligence is changing our world - it is on all of us to make sure that it works out". Our World in Data. Retrieved 8 October 2023.
^ Dickson, Ben (16 November 2023). "Here is how far we are to attaining AGI, according to DeepMind". VentureBeat.
^ McCarthy, John (2007a). "Basic Questions". Stanford University. Archived from the original on 26 October 2007. Retrieved 6 December 2007.
^ This list of smart qualities is based upon the topics covered by major AI textbooks, consisting of: Russell & Norvig 2003, Luger & Stubblefield 2004, Poole, Mackworth & Goebel 1998 and Nilsson 1998.
^ Johnson 1987.
^ de Charms, R. (1968 ). Personal causation. New York City: Academic Press.
^ a b Pfeifer, R. and Bongard J. C., How the body shapes the method we think: a brand-new view of intelligence (The MIT Press, 2007). ISBN 0-2621-6239-3.
^ White, R. W. (1959 ). "Motivation reassessed: The idea of competence". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ White, R. W. (1959 ). "Motivation reassessed: The idea of proficiency". Psychological Review. 66 (5 ): 297-333. doi:10.1037/ h0040934. PMID 13844397. S2CID 37385966.
^ Muehlhauser, Luke (11 August 2013). "What is AGI?". Machine Intelligence Research Institute. Archived from the original on 25 April 2014. Retrieved 1 May 2014.
^ "What is Artificial General Intelligence (AGI)?|4 Tests For Ensuring Artificial General Intelligence". Talky Blog. 13 July 2019. Archived from the initial on 17 July 2019. Retrieved 17 July 2019.
^ Kirk-Giannini, Cameron Domenico; Goldstein, Simon (16 October 2023). "AI is closer than ever to passing the Turing test for 'intelligence'. What occurs when it does?". The Conversation. Retrieved 22 September 2024.
^ a b Turing 1950.
^ Turing, Alan (1952 ). B. Jack Copeland (ed.). Can Automatic Calculating Machines Be Said To Think?. Oxford: Oxford University Press. pp. 487-506. ISBN 978-0-1982-5079-1.
^ "Eugene Goostman is a real young boy - the Turing Test states so". The Guardian. 9 June 2014. ISSN 0261-3077. Retrieved 3 March 2024.
^ "Scientists challenge whether computer 'Eugene Goostman' passed Turing test". BBC News. 9 June 2014. Retrieved 3 March 2024.
^ Jones, Cameron R.; Bergen, Benjamin K. (9 May 2024). "People can not differentiate GPT-4 from a human in a Turing test". arXiv:2405.08007 [cs.HC]
^ Varanasi, Lakshmi (21 March 2023). "AI designs like ChatGPT and GPT-4 are acing everything from the bar examination to AP Biology. Here's a list of tough examinations both AI versions have actually passed". Business Insider. Retrieved 30 May 2023.
^ Naysmith, Caleb (7 February 2023). "6 Jobs Artificial Intelligence Is Already Replacing and How Investors Can Profit From It". Retrieved 30 May 2023.
^ Turk, Victoria (28 January 2015). "The Plan to Replace the Turing Test with a 'Turing Olympics'". Vice. Retrieved 3 March 2024.
^ Gopani, Avi (25 May 2022). "Turing Test is undependable. The Winograd Schema is obsolete. Coffee is the answer". Analytics India Magazine. Retrieved 3 March 2024.
^ Bhaimiya, Sawdah (20 June 2023). "DeepMind's co-founder recommended evaluating an AI chatbot's ability to turn $100,000 into $1 million to determine human-like intelligence". Business Insider. Retrieved 3 March 2024.
^ Suleyman, Mustafa (14 July 2023). "Mustafa Suleyman: My brand-new Turing test would see if AI can make $1 million". MIT Technology Review. Retrieved 3 March 2024.
^ Shapiro, Stuart C. (1992 ). "Expert System" (PDF). In Stuart C. Shapiro (ed.). Encyclopedia of Artificial Intelligence (Second ed.). New York: John Wiley. pp. 54-57. Archived (PDF) from the initial on 1 February 2016. (Section 4 is on "AI-Complete Tasks".).
^ Yampolskiy, Roman V. (2012 ). Xin-She Yang (ed.). "Turing Test as a Specifying Feature of AI-Completeness" (PDF). Artificial Intelligence, Evolutionary Computation and Metaheuristics (AIECM): 3-17. Archived (PDF) from the original on 22 May 2013.
^ "AI Index: State of AI in 13 Charts". Stanford University Human-Centered Expert System. 15 April 2024. Retrieved 27 May 2024.
^ Crevier 1993, pp. 48-50.
^ Kaplan, Andreas (2022 ). "Artificial Intelligence, Business and Civilization - Our Fate Made in Machines". Archived from the original on 6 May 2022. Retrieved 12 March 2022.
^ Simon 1965, p. 96 quoted in Crevier 1993, p. 109.
^ "Scientist on the Set: An Interview with Marvin Minsky". Archived from the initial on 16 July 2012. Retrieved 5 April 2008.
^ Marvin Minsky to Darrach (1970 ), priced quote in Crevier (1993, p. 109).
^ Lighthill 1973; Howe 1994.
^ a b NRC 1999, "Shift to Applied Research Increases Investment".
^ Crevier 1993, pp. 115-117; Russell & Norvig 2003, pp. 21-22.
^ Crevier 1993, p. 211, Russell & Norvig 2003, p. 24 and see also Feigenbaum & McCorduck 1983.
^ Crevier 1993, pp. 161-162, 197-203, 240; Russell & Norvig 2003, p. 25.
^ Crevier 1993, pp. 209-212.
^ McCarthy, John (2000 ). "Reply to Lighthill". Stanford University. Archived from the initial on 30 September 2008. Retrieved 29 September 2007.
^ Markoff, John (14 October 2005). "Behind Artificial Intelligence, a Squadron of Bright Real People". The New York Times. Archived from the initial on 2 February 2023. Retrieved 18 February 2017. At its low point, some computer researchers and software engineers prevented the term expert system for fear of being considered as wild-eyed dreamers.
^ Russell & Norvig 2003, pp. 25-26
^ "Trends in the Emerging Tech Hype Cycle". Gartner Reports. Archived from the original on 22 May 2019. Retrieved 7 May 2019.
^ a b Moravec 1988, p. 20
^ Harnad, S. (1990 ). "The Symbol Grounding Problem". Physica D. 42 (1-3): 335-346. arXiv: cs/9906002. Bibcode:1990 PhyD ... 42..335 H. doi:10.1016/ 0167-2789( 90 )90087-6. S2CID 3204300.
^ Gubrud 1997
^ Hutter, Marcus (2005 ). Universal Expert System: Sequential Decisions Based on Algorithmic Probability. Texts in Theoretical Computer Technology an EATCS Series. Springer. doi:10.1007/ b138233. ISBN 978-3-5402-6877-2. S2CID 33352850. Archived from the original on 19 July 2022. Retrieved 19 July 2022.
^ Legg, Shane (2008 ). Machine Super Intelligence (PDF) (Thesis). University of Lugano. Archived (PDF) from the original on 15 June 2022. Retrieved 19 July 2022.
^ Goertzel, Ben (2014 ). Artificial General Intelligence. Lecture Notes in Computer Technology. Vol. 8598. Journal of Artificial General Intelligence. doi:10.1007/ 978-3-319-09274-4. ISBN 978-3-3190-9273-7. S2CID 8387410.
^ "Who created the term "AGI"?". goertzel.org. Archived from the original on 28 December 2018. Retrieved 28 December 2018., via Life 3.0: 'The term "AGI" was popularized by ... Shane Legg, Mark Gubrud and Ben Goertzel'
^ Wang & Goertzel 2007
^ "First International Summer School in Artificial General Intelligence, Main summertime school: June 22 - July 3, 2009, OpenCog Lab: July 6-9, 2009". Archived from the original on 28 September 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2009/2010 - пролетен триместър" [Elective courses 2009/2010 - spring trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the initial on 26 July 2020. Retrieved 11 May 2020.
^ "Избираеми дисциплини 2010/2011 - зимен триместър" [Elective courses 2010/2011 - winter trimester] Факултет по математика и информатика [Faculty of Mathematics and Informatics] (in Bulgarian). Archived from the initial on 26 July 2020. Retrieved 11 May 2020.
^ Shevlin, Henry; Vold, Karina; Crosby, Matthew; Halina, Marta (4 October 2019). "The limits of machine intelligence: Despite progress in maker intelligence, artificial general intelligence is still a major challenge". EMBO Reports. 20 (10 ): e49177. doi:10.15252/ embr.201949177. ISSN 1469-221X. PMC 6776890. PMID 31531926.
^ Bubeck, Sébastien; Chandrasekaran, Varun; Eldan, Ronen; Gehrke, Johannes; Horvitz, Eric; Kamar, Ece; Lee, Peter; Lee, Yin Tat; Li, Yuanzhi; Lundberg, Scott; Nori, Harsha; Palangi, Hamid; Ribeiro, Marco Tulio; Zhang, Yi (27 March 2023). "Sparks of Artificial General Intelligence: Early explores GPT-4". arXiv:2303.12712 [cs.CL]
^ "Microsoft Researchers Claim GPT-4 Is Showing "Sparks" of AGI". Futurism. 23 March 2023. Retrieved 13 December 2023.
^ Allen, Paul; Greaves, Mark (12 October 2011). "The Singularity Isn't Near". MIT Technology Review. Retrieved 17 September 2014.
^ Winfield, Alan. "Expert system will not develop into a Frankenstein's monster". The Guardian. Archived from the original on 17 September 2014. Retrieved 17 September 2014.
^ Deane, George (2022 ). "Machines That Feel and Think: The Role of Affective Feelings and Mental Action in (Artificial) General Intelligence". Artificial Life. 28 (3 ): 289-309. doi:10.1162/ artl_a_00368. ISSN 1064-5462. PMID 35881678. S2CID 251069071.
^ a b c Clocksin 2003.
^ Fjelland, Ragnar (17 June 2020). "Why general artificial intelligence will not be understood". Humanities and Social Sciences Communications. 7 (1 ): 1-9. doi:10.1057/ s41599-020-0494-4. hdl:11250/ 2726984. ISSN 2662-9992. S2CID 219710554.
^ McCarthy 2007b.
^ Khatchadourian, Raffi (23 November 2015). "The Doomsday Invention: Will expert system bring us utopia or destruction?". The New Yorker. Archived from the original on 28 January 2016. Retrieved 7 February 2016.
^ Müller, V. C., & Bostrom, N. (2016 ). Future progress in synthetic intelligence: A study of expert opinion. In Fundamental concerns of artificial intelligence (pp. 555-572). Springer, Cham.
^ Armstrong, Stuart, and Kaj Sotala. 2012. "How We're Predicting AI-or Failing To." In Beyond AI: Artificial Dreams, edited by Jan Romportl, Pavel Ircing, Eva Žáčková, Michal Polák and Radek Schuster, 52-75. Plzeň: University of West Bohemia
^ "Microsoft Now Claims GPT-4 Shows 'Sparks' of General Intelligence". 24 March 2023.
^ Shimek, Cary (6 July 2023). "AI Outperforms Humans in Creativity Test". Neuroscience News. Retrieved 20 October 2023.
^ Guzik, Erik E.; Byrge, Christian; Gilde, Christian (1 December 2023). "The originality of devices: AI takes the Torrance Test". Journal of Creativity. 33 (3 ): 100065. doi:10.1016/ j.yjoc.2023.100065. ISSN 2713-3745. S2CID 261087185.
^ Arcas, Blaise Agüera y (10 October 2023). "Artificial General Intelligence Is Already Here". Noema.
^ Zia, Tehseen (8 January 2024). "Unveiling of Large Multimodal Models: Shaping the Landscape of Language Models in 2024". Unite.ai. Retrieved 26 May 2024.
^ "Introducing OpenAI o1-preview". OpenAI. 12 September 2024.
^ Knight, Will. "OpenAI Announces a New AI Model, Code-Named Strawberry, That Solves Difficult Problems Step by Step". Wired. ISSN 1059-1028. Retrieved 17 September 2024.
^ "OpenAI Employee Claims AGI Has Been Achieved". Orbital Today. 13 December 2024. Retrieved 27 December 2024.
^ "AI Index: State of AI in 13 Charts". hai.stanford.edu. 15 April 2024. Retrieved 7 June 2024.
^ "Next-Gen AI: OpenAI and Meta's Leap Towards Reasoning Machines". Unite.ai. 19 April 2024. Retrieved 7 June 2024.
^ James, Alex P. (2022 ). "The Why, What, and How of Artificial General Intelligence Chip Development". IEEE Transactions on Cognitive and Developmental Systems. 14 (2 ): 333-347. arXiv:2012.06338. doi:10.1109/ TCDS.2021.3069871. ISSN 2379-8920. S2CID 228376556. Archived from the initial on 28 August 2022. Retri

הערות