The Udayagiri caves near Vidisha are around 14km (depending on route) from Sanchi, and we travelled there in a 3-wheeler, which was a nice ride through the fertile countryside. Although cave temples are mainly associated with Buddhism, in the later period Jainas and Hindus took to the idea also, but with a significant difference: their monastics do not seem to have lived in the caves; and in the case of Hindus, the caves were only used for shrines to the gods.

The Udayagiri caves are best known for their Jaina cave (no. 20), but when we visited the cave was closed because of the danger of collapse in the cave. The other caves, which were Hindu, were built starting around the 4th and 5th BCE. They contained reliefs of the mythic stories of the gods, and also shrines for various gods, particularly a large relief of the Varāha story in which Vishnu rescues the earth goddess Bhūdevī.

There were many other caves, spread over two hills which are around 1km from each other, and we spent a good couple of hours at the site, which had no other visitors. Mainly the site was well looked after, and was quite clean, and had only a little graffiti. There was no charge to go in, but we tipped the caretaker for showing us round, and explaining many of the sights.

The Udayagiri Caves feature some of the oldest Hindu images and cave temples in India. They are located near the city of Vidisha, northeast of Bhopal in the state of Madhya Pradesh, around 14km from the Buddhist site at Sanchi. One of India's most important archaeological sites from the Gupta period, the Udayagiri hills and its caves are an archaeological site under the protection of the Archaeological Survey of India (ASI).

Udayagiri consists of two low hills immediately next to the River Bes. Located a short distance from the earthen ramparts of the ancient city site of Besnagar. Udayagiri is best known for a series of rock-cut sanctuaries and images excavated into hillside in the early years of the fifth century CE. The site is notable for its ancient monumental relief sculpture of Hindu god Vishnu, in his incarnation as the boar-headed Varāha, rescuing the earth symbolically represented by Bhūdevī clinging to the boar's tusk as described in Hindu mythology.

The site has important inscriptions of the Gupta dynasty belonging to the reigns of Chandragupta II (c. 375-415) and Kumaragupta I (c. 415-55). In addition to these remains, Udayagiri has a series of rock-shelters and petroglyphs, ruined buildings, inscriptions, water systems, fortifications and habitation mounds, all of which have been only partially investigated. The complex consists of twenty caves, of which one is dedicated to Jainism, and all others to Hinduism.

The site at Udayagiri Caves was extensively reworked under the patronage of Chandragupta II, who ruled the Gupta Empire between c. 380 and 413/415 CE. Archaeologist Michael D. Willis argued that Chandragupta II did so in order to reflect a new concept of Hindu kingship, in which the monarch was seen as both the paramount sovereign (cakravartin) and the supreme devotee of the god Vishnu.

Cave 5: Vishnu as Varāha, showing the god controlling the waters, personified as a serpent (nāga), and carrying the earth goddess on his tusk, cave 5 is a shallow niche more than a cave and contains the much-celebrated figure of Vishnu in his Varāha or Boar-headed incarnation. The complex iconography of the tableau has been explained by Debala Mitra. Willis has described the relief as the "iconographic centre-piece of Udayagiri".

Cave 6 is directly beside Cave 5 and consists of rock-cut cella entered through an elaborate T-shaped door. The original image inside is missing but it was probably a Shiva linga. Outside the cave is a panel with an inscription recording the creation of the 'meritorious gift' (deyadharma), probably the cave and the adjacent images, in Gupta year 82 (401 CE). In the ceiling of the cave is an undated pilgrim record of somebody named Shivāditya.

The door guardians flanking the entrance are regarded by art historians as among the most powerful works of early Gupta sculpture. Beside them, on either side, are figures of Vishnu and of Shiva Gangadhara, the latter much worn from the falling of water over the image. Of special note is Durga slaying the Buffalo Demon, one of the earliest representations of the theme in India. Of special note also is the figure of seated Ganesh, to the left of the cave entrance, and the rectangular niche with seated goddesses, located to the right. Aside from this being the oldest datable Ganesh in India, the arrangement, with a guarded sanctum in the centre, Ganesh on one side and the mother goddesses on the other, presages the arrangement of temple space in subsequent centuries.

Cave 4 has a rectangular cella with a rock-cut plinth in which is set a spectacular Shiva linga. The hair is tied up into a topknot with long locks cascading down each side. The arrangement of the hair recalls the story of how Shiva broke the fall of the River Ganga as the waters came down from heaven. There is a water channel in the plinth and in the floor of the chamber leading to a hole that pierces in the cave wall. The cave is entered through an entrance of exquisite proportions with delicately carved floral scrolls. The lintel of the door extends beyond the jambs to create a T-shape, a common characteristic of early temple architecture. Unlike most doors, however, the frame consists only of square moulding, identical on the top and sides. The base of the jambs and the sill are modern replacements. Externally, the cave is flanked by rock-cut pilasters and two guardians (dvārapāla) now damaged and weather-worn.

The Passage, which starts beside Cave 8, is a unique feature of Udayagiri. It consists of a natural cleft or canyon in the rock running approximately east to west. The passage has been subject to a series of modifications and additions, the sets of steps cut into the floor being the most conspicuous feature. The lowest set of steps on the right hand side is visibly water-worn and evidently served as a water-cascade in historic times.

Shell inscriptions engraved on the upper walls of the passage are the largest examples of this kind of writing known in India. The images of the fifth century cut through the Sankha Lipi indicating they pre-date Gupta times. The inscriptions, which appear to be names in Sanskrit, had not been fully deciphered until recently. The upper walls of the passage have large notches at several places, indicating that stone beams and slabs were used to roof over parts of the passage, giving it a significantly different appearance from what can be seen today. In terms of sculpture, the passage has a series of niches and caves, numbered 9 through 14. Only a few contain sculptures, mostly of standing Vishnu, all of which are damaged.

Cave 13 contains a large figure of Narayana, the recumbent figure of Vishnu resting. Before the niche are two shallow recess in the floor. These received pillar bases for some sort of porch. There is a shallow vertical recess above the cave, matched by a similar recess in the opposite cliff face, suggesting that there was some sort of architectural curtain wall across the passage at this point. Beside the image of Narayana is a kneeling devotee, and it has been argued that this figure is a depiction of Chandragupta II himself, symbolising his devotion to Vishnu.