Dhenu Vision: a multimodal model for the field

We’re releasing Dhenu Vision, a multimodal model for working with crop photographs. The model card is on Hugging Face as KissanAI/Dhenu-vision-lora-0.1. This post is an engineering note on what is inside the model, what failure modes it has, and what the use case is for a farmer who will eventually use it through one of our deployment partners.

What it does

A farmer photographs a leaf. The model identifies the crop, the visible symptom, and a likely class of cause such as pest, disease, or nutrient stress. It then connects that observation with the farmer’s question and relevant agronomic context. A disease label by itself is not very useful in the field. The answer also needs an explanation and a sensible next step.

Dhenu Vision is not a standalone product-registration or legal-recommendation system. Product-level guidance requires current local information and an agronomy workflow outside the vision model itself.

The initial release covers three major crops in depth (rice, maize, and wheat), with ten common diseases and pests per crop. This is deliberately narrow. We could have shipped something broader and weaker; we chose to ship narrower and stronger because in crop protection, a wrong diagnosis at scale is worse than no diagnosis.

The model is voice-callable through the same Dhenu pipeline as our text models. A farmer can speak a question alongside the photograph (“the leaves of my rice are turning yellow at the tips, what is happening?”) and get an answer back in the same language, grounded in both the image evidence and the question.

How it is built

Dhenu Vision is a vision-language LoRA fine-tune over Qwen-VL-Chat as the base model. The choice of base was deliberate. Qwen-VL handles non-English text in image captions and queries better than the LLaVA family, which matters because the prompts we feed the model often include Indic-script terms (a Marathi-named pest, a Tamil crop variety).

The training material comes from published plant-pathology research datasets and agricultural extension publications, followed by substantial curation from our team. Finding an image is the easy part. The work is pairing it with the right crop context, question, answer, and reasoning so the model learns to explain what it sees rather than simply attach a label.

Each image is tagged with more than a disease name: the crop, the growth stage at which the symptom appeared, the agro-climatic zone of origin, the typical severity classification, and the recommended action class. The downstream assistant needs those fields to suggest an appropriate intervention. A disease name alone does not get it there.

The base model stayed frozen during the LoRA pass. As with most domain models, the difficult part was not the final compute run. It was the agronomy review and data curation required to turn inconsistent source material into useful multimodal examples.

Where it falls down

These are the failure modes we are explicit about.

Disease look-alikes the eye cannot disambiguate. Some bacterial and fungal symptoms on the same crop look identical to the camera. The model will sometimes call one when the answer is the other. The action plans in those cases are deliberately conservative: when the model’s confidence is below threshold on a fungal-vs-bacterial call, the recommendation asks the farmer to take a second photograph in better light or to escalate to a human agronomist. We would rather be cautious than confidently wrong.

Rare regional pests. A pest that is endemic in one state but rare elsewhere will be undertrained. The model will tend to default to the more common look-alike. We address that gap by adding more regionally balanced examples through the same curation and review pipeline.

Late-stage symptoms. A crop with advanced disease damage looks like a different problem than the same crop in early-stage symptom presentation. The model is stronger on early to mid-stage than on late-stage. The product implication is to encourage farmers to photograph at first sign rather than after damage has progressed, which is also the agronomically correct behaviour.

Comparison to general vision models on agronomy failure modes

We compared Dhenu Vision with the general vision models available at the time on the same kinds of field photographs. The detailed evaluation belongs with the model artifacts, but the qualitative result was consistent: a domain model was more reliable on the crop and symptom families it had been curated to understand.

The general models know what bacterial leaf blight is. They do not always recognise it in a photograph taken in afternoon sun on an older phone camera, especially when the crop, region, and farmer’s description are missing from the prompt. Dhenu Vision benefits from having those field conditions represented in its curated examples.

The same distinction matters after identification. The vision model can contribute the visual evidence and likely diagnosis class, but a deployed assistant still needs current agronomic context and a cautious escalation path. That separation is deliberate. The image model should not pretend that recognising a symptom is the same thing as making a complete field decision.

What we are publishing and why

The model card and LoRA adapter are available through the KissanAI organization on Hugging Face. The most reusable part of the work is the methodology: connect each image to the crop context, the farmer’s question, a reviewed answer, and the reasoning that links the evidence to the answer.

The reason to release this is the same reason we released Dhenu 1.0: other teams should be able to build on, evaluate against, and improve this work, and the parts of KissanAI that are hard to copy were never the adapter weights.

Next we are adding crops, improving the regional balance, and working through the failure modes above. If you build on the adapter, send us the photographs it gets wrong. Those are more useful than a general thumbs-up.