The Moon’s cratered surface may seem like the perfect place to hand repetitive work to machines. There are millions of impact marks to track, and every one of them can help tell time on a planetary scale. But a new analysis argues that many AI-built lunar crater catalogs are not yet as reliable as their headline numbers suggest.
In planetary science, crater catalogs do far more than list circles on a map. They record location, size, and other characteristics of impact structures, giving researchers a way to estimate surface ages, reconstruct geologic history, and study how landscapes changed over time. If those measurements are wrong, the science built on top of them can slip as well.
“AI has enormous potential to help with repetitive, time-consuming scientific tasks, especially gathering some of our data,” said Dr. Stuart J. Robbins of Southwest Research Institute’s Solar System Science and Exploration Division in Boulder, Colorado, who led the study. “But our analysis shows that researchers should not assume an AI-generated crater catalog is ready for scientific use solely based on its published metrics.”
Robbins and co-author Dr. Rachael H. Hoover compared eight lunar crater databases produced with automated methods, including machine learning systems and earlier non-ML approaches. They tested each one against a large manually compiled lunar catalog assembled by Robbins over several years, then evaluated them using the same crater-matching rules across the board.
That detail matters because performance can change sharply depending on what counts as a successful crater “match.” For many uses in planetary science, a crater is only useful if it is both in the right place and sized correctly. A system can appear strong under broad computer-vision measures while still being off enough to create trouble in real scientific work.
“A crater catalog is not just a random list of circles,” Robbins said. “If a crater is shifted, duplicated or improperly sized, that can affect the science that depends on those metrics. For instance, if a surface with a model age of 1 million years requires x number of craters and AI accidentally duplicates those craters, suddenly the model would double the surface’s projected age.”
The team found that many published metrics fell once the databases were tested under stricter, uniform standards. Using criteria based on the repeatability of manual crater analysts, nearly all of the automated catalogs performed worse than their published values suggested. In some cases, the drop was more than a factor of 10.
That does not mean every catalog failed in the same way. Some worked reasonably well over certain crater sizes and much worse at others. A single overall score could make a database look acceptable even when its usefulness changed sharply across the diameter ranges that matter most to researchers.
“Diameter dependence matters,” Robbins said. “A catalog might look acceptable from one overall number, but when you break it down by crater size, it may be useful for one question while unreliable for many others.”
Crater counting is one of the basic tools scientists use to date solid worlds. Small asteroids hit the Moon and other bodies at a roughly steady pace, so surfaces with more craters are generally older than surfaces with fewer. Researchers compare crater sizes and densities, then use models of impact rates to estimate the age of terrain.
That is why these databases are so important. Impact cratering is the most common surface process across the solar system’s rocky worlds, and crater catalogs support work on age modeling, geologic reconstruction, and crustal properties. They have traditionally been built by hand or with semiautomated methods, a slow process that can take years and still carries some subjectivity.
AI and machine learning promise a way out of that bottleneck. With better automation, scientists could process enormous datasets faster and study crater populations at scales that would be difficult for people alone. The new study does not reject that goal. Instead, it argues that the field needs clearer rules for deciding when an automated catalog is actually good enough to trust.
To test that, the authors compared eight large-coverage lunar crater databases: Salamunićcar et al. 2014, Wang et al. 2015, Silburt et al. 2019, Yang et al. 2020, Wang et al. 2021, La Grassa et al. 2025a, La Grassa et al. 2025b, and Xiong et al. 2025. They measured them against the manually compiled Robbins 2019 reference database, which the study says is approximately complete for craters at least 1 to 2 kilometers across.
The paper also pushes back on the heavy use of intersection over union, or IoU, as a default score. That metric is common in computer vision, but the authors argue it is poorly suited to impact craters because it can accept a crater whose overlap looks decent even when its diameter or position is inaccurate enough to distort scientific analysis.
The broader problem, the authors argue, is inconsistency. Different teams use different definitions of a match, different tolerance levels for size and location, and often do not explain those choices clearly. At the same time, many users may treat published precision and recall values as proof that a catalog is ready for science without doing independent validation.
The new comparison suggests that confidence is often misplaced. The study reports that most of the AI-based databases showed size and location biases, and that their spread in measurements was generally worse than the spread seen among expert human crater analysts. In practical terms, that means ages inferred from those craters could carry larger uncertainties and biases than standard error models might imply.
The authors also found no clear trend showing that newer AI crater catalogs have become steadily better under common, uniform testing. One of the stronger-performing datasets in the comparison came from 2014 and relied on deterministic AI methods rather than modern machine learning, though it also included manual checking of every feature.
Hoover said the point is not to push AI out of planetary science but to make its use more rigorous. “Our work highlights the necessary next step of standardizing benchmarks, including transparent reporting of matching criteria and independent validation, so AI-generated catalogs can be properly used for scientific analysis,” she said.
Robbins put it more plainly. “AI may eventually transform crater cataloging and revolutionize how we gather our science data, potentially saving years of time,” he said. “For now, researchers need to not chase it as the solution to everything. We need to understand how these tools work, where they fall short and whether their performance is good enough to support the science being done.”
For planetary scientists, the message is caution, not retreat. Automated crater catalogs may still be useful as a starting point for manual screening or for research questions that can tolerate lower accuracy. But the study argues they should not be treated as interchangeable with expert-built catalogs simply because they report strong headline metrics.
The authors recommend clearer reporting of matching tolerances, independent checks against reference databases, and precision and recall values broken down by crater size.
They also suggest that future AI work may help most by focusing on smaller craters, where manual cataloging is hardest and where current systems still struggle.
Research findings are available online in The Planetary Science Journal.
The original story “AI can quickly spot and catalog lunar craters, but accuracy remains a major problem” is published in The Brighter Side of News.
Like these kind of feel good stories? Get The Brighter Side of News’ newsletter.
The post AI can quickly spot and catalog lunar craters, but accuracy remains a major problem appeared first on The Brighter Side of News.
Leave a comment
You must be logged in to post a comment.