Because of the nature of human consciousness. In a sense, the science of relativity is 'occult' (hidden) to anyone not savvy with higher math. Get savvy and one can sit down and follow the dots pretty well. Same with the spiritual (a word akin to 'ufo' - it has so many delineations and shades of meaning - though like 'ufo' it's overall meaning cruises at a particular altitude, enough so to be predictable). Just like the neophyte mathematician must 'initiate' themselves into higher math to have the ability to understand and experience the mathematical universe, so too the spiritual seeker must initiate themselves into higher realms of consciousness in order to understand and experience the subtle aspects of existence. That knowledge of the subtle realms is 'occult' - it is 'hidden' from the 'uninitiated'.
There's another layer: since in progressing into deeper and deeper aspects of the human and the world (via inner exercises - a kind of spiritual heavy-lifting, like body building, but on a spiritual level) one gains access to forces and powers, develops capacities, that are objectively real but are not in themselves endowed with 'goodness'. How to use such capacities and powers - for good or ill - is in the realm of the free will of the 'occultist', the 'magician', the 'initiate', the spiritual worker. Hence, there can be those who have achieved capacities that then use the capacities for ill - and those that do so, will have a reckoning - but until then the 'dark magician' can do much mischief if not outright evil. In fact, such decisions to work for the self will cause the door to the spiritual worlds to be shut and bring down consequences on the occultist.
The above is not as alien as it may sound - for the same is true for the scientist. The scientist works objectively and what s/he does with the knowledge s/he gains is in the realm of free will. Great good and great ill can be the chosen path - with the consequences.
Same for the spiritual seeker, the 'occultist' - except what is accessed is via the actual instrument of the human being - far more intimate and 'close to the bone'. Such knowledge - when accessed - has 'laws' that if not respected cause havoc.
A story: in occultism - but I will use the word 'esotericism' (less threatening and 'loaded') - there are exercises to train the Will. Now when you do these exercises under guidance you are doing them in conjunction with exercises pertaining to Love and Thought. The guide is monitoring. Were one to pursue just one exercise one would get a lop-sided development - as took place in the instance I will describe. There was a South American dictator whose cruelty was epic. His ability to inflict such savagery, at the scale achieved, was rooted in a powerful Will. The dictator was interviewed and asked where he felt he had gotten his exceptional Will that enabled him to press his agenda - which was a dark and evil agenda - against all odds with such force and consistency. His answer was that when in jail early in his career he was given an esoteric text of occult exercises (in common parlance we would call such self-development exercises, except these were called otherwise). He chose the exercise pertaining to the Will - ignoring the injunctions in the occult text against one-sided development. The result was that the man's Will became a force of great power - though his capacity for compassion and ethical thinking must have shriveled up in the face of the power he accessed with a plumped up Will. (Without purifying his emotional self/body - his Will became driven by his lower desires and appetites - albeit in a disciplined way). His choices wound up being for power - and his augmented Will sustained that urge to power to the point of barbarity. The dictator was very open about the fact that he ascribed his success to the Will exercise of this particular occult teacher whose book he had read.
Hence why it has been in the past that such exercises and such inner development has been 'hidden' so that such aberrations of development do not occur. Hence why it is that 'occult knowledge' - hidden knowledge - has been feared. Fact is, life can bring you to points of initiation. Life can teach compassion, or hone the intellect, or school the Will. At a certain point conscious development must be undertaken - and to do that one must access the 'hidden' pathways to the higher states of being. No one is barring the way - no one is burying books - it's not that kind of knowledge. Access is wholly dependent on the person - just as access to the mathematical universe is dependent on study.
