An LLaMA 3.1-Based Chatbot with Retrieval-Augmented Generation (RAG) for Academic Services at UPN “Veteran” Yogyakarta

Introduction

The rapid development of artificial intelligence (AI) and natural language processing (NLP) technologies has significantly transformed how users interact with digital information systems. In particular, the emergence of Large Language Models (LLMs) has enabled machines to understand and generate human-like text, facilitating the development of intelligent conversational agents capable of assisting users in various domains (1, 2). In higher education environments, access to accurate and timely academic information is essential for supporting student activities such as course registration, curriculum planning, academic regulations, and thesis administration. However, many academic information services in universities still rely on static websites or manual administrative communication, which may limit accessibility and responsiveness. These limitations often prevent students from obtaining information efficiently, particularly outside office hours. As digital transformation accelerates within educational institutions, there is an increasing demand for intelligent systems capable of providing real-time, automated academic assistance to students (3, 4).

Recent studies have explored the use of chatbots powered by LLMs to improve information accessibility and user interaction in educational environments (5, 6). These systems can process natural language queries and generate contextually appropriate responses, allowing students to access academic information more efficiently (7). Nevertheless, conventional LLM-based chatbots still face a critical limitation known as hallucination, where the model generates responses that appear plausible but are factually incorrect because they rely solely on pretrained knowledge without access to external data sources (8). This issue becomes particularly problematic in domains such as academic administration, where accurate and reliable information is essential.

To address this limitation, researchers have proposed Retrieval-Augmented Generation (RAG), an architecture that integrates information retrieval with generative language models (9, 10). In this approach, relevant documents are first retrieved from a knowledge base and then provided as contextual input to the language model to generate more accurate and evidence-grounded responses (11). The RAG framework has been shown to significantly improve factual accuracy, contextual relevance, and reliability in knowledge-intensive tasks compared with traditional LLM systems (12–14).

Despite these advances, standard RAG architectures utilizing a single retrieval stream struggle in specialized academic environments. Academic documents possess a unique structural hierarchy and semantic density, containing exact codes (e. g., course codes like “123210052”), rigid numeric rules (e. g., “minimum 130 SKS” or “GPA of 2.00”), and specific nomenclature (e. g., lecturer names with intricate academic titles) alongside general academic jargon. Standalone semantic search utilizing dense vector embeddings often misses exact lexical matches for numeric constraints and course codes due to embedding compression. Conversely, standalone keyword-matching search (such as BM25) fails to capture semantic synonyms or conversational student phrasing (e. g., mapping “bimbingan skripsi” to “konsultasi tugas akhir”). Thus, a critical research gap exists regarding how to establish a robust local RAG system that balances lexical precision and semantic flexibility within localized, resource-constrained higher education settings.

Furthermore, to the best of our knowledge, most existing RAG-based chatbot studies for academic services focus on English-language corpora. This study addresses an underexplored gap by validating a hybrid RAG architecture on Indonesian-language institutional documents, where mixed formal-informal language patterns and domain-specific acronyms (e. g., SKS, MBKM, KRS) present distinct retrieval challenges not addressed by prior work.

Alongside the development of retrieval-augmented systems, recent progress in open-source LLM architectures has enabled institutions to implement advanced AI systems locally. One of the most influential models is the LLaMA family of language models, which offers competitive performance while allowing flexible deployment for domain-specific applications (15). Integrating LLaMA models with RAG architectures provides a promising approach for developing domain-specific conversational systems that utilize institutional knowledge sources effectively (16). Despite these advances, the implementation of RAG-based chatbots for academic service environments remains relatively limited, particularly in the context of integrating modern LLM architectures with hybrid retrieval mechanisms to improve information accuracy and accessibility. Therefore, this study aims to develop and evaluate an academic chatbot system using the LLaMA 3.1 large language model integrated with a Retrieval-Augmented Generation (RAG) framework to support academic information services for Informatics students at Universitas Pembangunan Nasional “Veteran” Yogyakarta. The proposed system utilizes institutional academic documents as its knowledge base and applies hybrid retrieval methods combining BM25 and semantic search to enhance information retrieval before response generation. The system architecture consists of a document store, a chatbot engine integrated with LLaMA through Ollama, a user interface built with Streamlit, and an evaluation module designed to measure system performance using multidimensional metrics such as context relevance, answer faithfulness, and response accuracy. Through this approach, the study aims to improve the accuracy, efficiency, and accessibility of academic information services while demonstrating the practical application of RAG-based LLM systems in higher education environments.

This study addresses these gaps by implementing and evaluating a locally deployed academic chatbot system using the LLaMA 3.1 large language model integrated with a hybrid RAG framework at Universitas Pembangunan Nasional “Veteran” Yogyakarta. The system runs completely locally to preserve data privacy and eliminate operational costs. We propose a hybrid retrieval strategy that combines BM25 lexical search with dense vector embeddings utilizing a multilingual semantic model, resolved through max-score linear scaling. Furthermore, we implement an adaptive category-specific prompt engineering mechanism to handle the diverse inquiries of informatics students. This paper presents a multidimensional evaluation framework comprising RAG-specific criteria (via RAGAS), traditional NLG metrics, robustness assessments, system latency profiling, and exploratory usability testing to validate the engineering choices and empirical performance of the local RAG architecture.

Methodology

Research Design

This study employed an experimental system development approach to design and evaluate an academic service chatbot for students of the Informatics Study Program at UPN “Veteran” Yogyakarta. The development process followed the Rapid Application Development (RAD) framework, emphasizing iterative prototyping and continuous refinement of system functionality. This approach enabled rapid integration between the retrieval component and the generative language model while ensuring the system could respond effectively to students’ academic queries (17).

Data Sources and Preparation

The knowledge base was constructed using 10 official academic PDF documents obtained from the Informatics Study Program at UPN “Veteran” Yogyakarta, spanning academic guidelines, curricula, internship guidelines, and thesis regulations. Data preparation involved robust text extraction, cleaning, and normalization to remove formatting artifacts.

Following extraction, the text was segmented into smaller units using a RecursiveCharacterTextSplitter. To preserve semantic cohesion, adaptive chunking configurations were applied based on document category, as detailed in Table 1.

The segmentation pipeline yielded 263 high-coherence text chunks. Each chunk was indexed in parallel into two vector stores: a lexical store via BM25 and a semantic vector store utilizing the intfloat/multilingual-e5-small embedding model (384 dimensions) (18).

Hybrid Retrieval Mathematical Framework

To query the document store, the system implements a hybrid retrieval framework that dynamically merges keyword matching and vector-space similarities. Let Q represent the student’s query and D represent a document chunk within collection C.

Lexical Component (BM25)

The lexical relevance of document D to query Q is computed using the BM25Okapi formula, with hyperparameters k₁ = 1.2 and b = 0.75 (standard defaults). This score captures exact token-level matches, making it particularly effective for queries containing precise course codes, acronyms, and numeric thresholds.

Semantic Component

Semantic similarity is calculated as the cosine similarity between the 384-dimensional dense vector representations of the query and the document chunk, generated by the multilingual-e5-small embedding model (18). This score captures conceptual relatedness and handles natural language variation in student queries.

Normalization and Linear Weight Fusion

Because lexical scores produced by BM25 are unbounded ([0, ∞)) and semantic scores are bounded cosine similarities ([−1, 1]), direct addition is mathematically invalid and would skew results. To address this, Maximum Score Linear Scaling is applied to normalize BM25 scores as seen in Equation 1. Once normalized, both scores occupy a consistent [0, 1] range. The final hybrid score was calculated using the Equation 2.

In this study, the mixing parameter was set to α = 0.3, allocating 30% weight to keyword matching (preserving precise codes and names) and 70% weight to semantic understanding (providing tolerance for natural variations in student phrasing). This weighting reflects the structural characteristics of the target corpus: while the documents contain entity-critical terminology requiring lexical anchoring, the predominant content consists of procedural descriptions and regulatory clauses that benefit more from semantic comprehension. This is consistent with findings demonstrating that dense semantic retrieval outperforms sparse lexical methods for natural-language queries (19), while the retained BM25 component prevents systematic failures on entity-dense queries (9). The top K (K = 10) chunks with the highest final scores are selected as the context window.

Generative Modeling and System Configuration

The generation module utilizes the LLaMA 3.1 large language model (8B parameter chat-instruct version), running locally through Ollama (20). To reduce hallucination, we implemented an adaptive prompt engineering mechanism. Depending on the classified query intent (e. g., “dosen”, “kurikulum”, “prosedur”), the system dynamically attaches specialized instructions to the system prompt. Crucially, the prompt is structured with a rigid negative constraint: if the retrieved context window does not contain information matching the student's query, the model is strictly instructed to state that the information is unavailable in the database rather than extrapolating or guessing from its parametric memory.

The model parameters were configured to prioritize factual determinism: temperature = 0.3, top_p = 0.9, top_k = 40, and the context window size (num_ctx) was restricted to 6144 tokens to prevent context dilution. The complete system was implemented in Python using LangChain as the orchestration framework, with the intfloat/multilingual-e5-small model from the Sentence Transformers library for embedding generation. Document vectors were indexed using FAISS (Facebook AI Similarity Search). The chunking parameters (chunk size and overlap per document category as shown in Table 1), retrieval top-K setting, and prompt templates are fully documented to support reproducibility.

Results and Discussion

Ablation Study of Retrieval Configurations

To evaluate the proposed hybrid retrieval configuration, we performed an ablation study by isolating individual components of the retrieval pipeline. Standard standalone lexical search (BM25 Only) and standalone semantic search (Semantic Only) were evaluated against the combined Hybrid (30: 70) configuration across 22 validated academic test cases. The performance was analyzed using the RAGAS framework (21) and is summarized in Table 2. Due to computational constraints of the local RAGAS evaluator running on consumer-grade hardware, the RAGAS framework metrics (faithfulness, answer relevancy, context precision, and context recall) were computed as batch-level aggregate scores across all 22 test cases simultaneously, consistent with the reference implementation by Es et al. (21).

The hybrid retrieval method achieved the highest faithfulness score (0.712), outperforming both the BM25-only (0.550) and semantic-only (0.536) baselines. This result aligns with the theoretical framework established by Lewis et al. (9), who demonstrated that response faithfulness in RAG systems is fundamentally constrained by retrieval completeness: when retrieved chunks fail to capture all relevant information, the LLM is compelled to supplement with parametric knowledge, thereby increasing hallucination risk. The hybrid approach, by achieving the highest Context Recall (0.895), provides the LLM with a more complete contextual grounding, directly translating to superior faithfulness which is a causal mechanism consistent with the retrieval-generation dependency described by Borgeaud et al. (11).

Standard semantic retrievers sometimes fetch contextually related but factually incorrect pages, whereas keyword-matching often fails to capture the overall meaning. Combining them ensures that the generated responses are well-supported by the extracted source text. This is particularly critical in the academic domain where precise regulatory information (SKS thresholds, credit requirements, procedural deadlines) must be reproduced accurately.

Resolving the Hybrid Latency Paradox

A noteworthy technical observation in Table 2 is that the Hybrid approach (7.64 seconds) is significantly faster than the Semantic Only approach (13.23 seconds), despite executing both retrieval streams. Beyond the mean latency, the distribution reveals important characteristics: the standard deviation of 4.15 seconds (range: 0–21.44 seconds) reflects that simple factual queries (e. g., “Who is the head of department?”) resolve in under 6 seconds, whereas complex multi-step procedural queries require up to 21.44 seconds due to longer generation sequences.

This performance behavior is due to two critical engineering factors:

Parallel Query Execution: The BM25 search and semantic embedding generation are processed asynchronously in separate threads, minimizing pipeline blockages.

Aggressive Context-Size Optimization: In the Semantic Only configuration, similarity matching must calculate cosine similarities across the entire high-dimensional vector space, which is computationally expensive on local consumer-grade CPUs. The Hybrid configuration utilizes the rapid, linear-time BM25 pipeline as a coarse pre-filtering heuristic, pruning irrelevant segments quickly and reducing both context length and downstream token generation time of the LLaMA model.

Evaluation of Generated Response Quality (NLG Metrics)

To verify the linguistic accuracy of the generated responses against human-validated ground truths, we evaluated the output using standard Natural Language Generation (NLG) metrics across 22 test cases. The results are detailed in Table 3.

Per-Category Performance and Error Analysis

The mean Token F1-score of 0.499 and BLEU score of 0.233 are moderate values that require contextual interpretation. In academic chatbot systems, lower lexical overlap does not necessarily indicate a lack of accuracy. Our analysis identified two main contributing factors:

Syntactic Synthesis and Paraphrasing: The LLaMA 3.1 model presents responses in natural, polite Indonesian with structured formatting (bulleted lists, headings), whereas the ground-truth references are concise text extracted directly from administrative PDFs. Lexical metrics penalize this elaboration even when the content is factually correct.

Granular Vocabulary Divergence: For example, a query about thesis seminar registration procedures generated a detailed, formatted multi-step response with clothing requirements and time instructions (Token F1: 0.261, BLEU: 0.009), yet the response was factually accurate (faithfulness: 0.712) and received high user satisfaction. This confirms that lexical metrics are insufficient for evaluating generative systems where elaboration is expected (8).

The evaluation framework also tracked a Mean Robustness Score of 0.467. This metric quantifies the model's structural resilience when handling non-standard student inputs, lexical variations, and minor typographical errors within the Indonesian academic domain. A score of 0.467 indicates that while the local system successfully circumvents catastrophic behavioral failure or hallucination under linguistic noise, localized variations in semantic alignment still occur due to the rigid constraint boundaries of the underlying document corpus. Per-category analysis further reveals a fairness gap across information domains as seen in Table 4.

The Token F1 fairness gap (min: 0.325 for Tugas Akhir, max: 0.891 for Dosen) is attributable to structural differences across information types, not retrieval failure. Lecturer data, presented as structured name-title pairs, achieves near-perfect lexical overlap. In contrast, thesis and curriculum categories require multi-document synthesis and naturally produce paraphrased, elaborated responses. This finding is consistent with Ji et al. (8) who noted that lexical overlap metrics are insufficient for evaluating generative systems where answer elaboration and paraphrase are expected behaviors.

User-Based Exploratory Usability Testing

To assess user satisfaction, an exploratory usability pilot study was conducted with five informatics students from UPN “Veteran” Yogyakarta (active second-semester and eighth-semester students). The small-scale pilot focused on identifying immediate usability issues and validating the interface. Given the limited sample size, these results are explicitly exploratory in nature and should be validated with a larger and more diverse participant pool in future work. Respondents interacted with the system and rated performance on a Likert scale (1 to 5). The aggregated results are summarized in Table 5 (raw individual-level scores are provided in Appendix A).

The pilot evaluation indicated high user acceptance. The highest rating was achieved in Language Naturalness (4.80), confirming the effectiveness of LLaMA 3.1’s instruction-following capabilities. The high score for Response Speed (4.79) reflects the real-time usability of the local deployment.

Security, Privacy, and Ethical Considerations

A critical advantage of the proposed local RAG architecture is its alignment with data privacy regulations and security standards. Standard cloud-based chatbot implementations rely on external APIs (such as OpenAI GPT or Anthropic Claude), which require transmitting query logs to third-party servers. In higher education, sending query logs externally can pose risks to student privacy and violate institutional data handling policies.

By deploying LLaMA 3.1 locally via Ollama on institutional hardware, the following benefits are achieved:

Data Containment: Student queries, conversation histories, and internal academic documents remain entirely within the university’s local network infrastructure.

Mitigating Malicious Exploitation: Local deployment prevents external telemetry leaks and allows the university to implement strict rate-limiting and custom input-filtering heuristics to prevent adversarial injection and prompt-jailbreaking.

No External API Dependencies: Running the system locally avoids cost overheads and limits downtime risks associated with external API failures, making it a reliable and sustainable option for public universities.

Limitations and Future Outlook

While the hybrid RAG chatbot demonstrates strong performance, we acknowledge several engineering limitations and areas for future development:

Scale of Evaluation and Pilot Usability Testing: The dataset was restricted to 10 administrative PDFs, and user testing was limited to a pilot study of five respondents. Although the documents covered 100% of the active regulations for the department at the time of the study, expanding the knowledge base to encompass university-wide data and conducting larger-scale usability evaluations will be necessary for broader deployment.

Baseline Retrievers: The evaluation focused on standard BM25 and semantic search. Future iterations will evaluate modern dense retrievers (such as ColBERT or cross-encoder rerankers) to determine if they can further improve context precision without introducing significant latency.

Handling Complex PDF Formats: Like many standard RAG pipelines, our text splitter occasionally struggles with multi-column tables, diagrams, and scanned PDFs. Integrating advanced optical character recognition (OCR) and layout-aware table parsers (such as LayoutLM or TableTransformer) would help preserve the structure of complex academic documents.

Informal Language Handling: User feedback indicates the system is not yet fully optimized for handling student abbreviations and non-standard terms, which can affect retrieval quality and should be addressed through query expansion techniques in future work.

Conclusion

This study demonstrates that integrating the LLaMA 3.1 large language model with a Retrieval-Augmented Generation (RAG) framework is an effective approach for providing automated academic services in higher education. By combining BM25 lexical search and semantic vector similarity through a hybrid retrieval strategy with max-score linear scaling, the system achieved a faithfulness score of 0.712 and a context recall of 0.895, outperforming standalone lexical and semantic retrieval baselines. Local deployment via Ollama ensures complete data privacy and eliminates operational costs, while exploratory user testing produced a high satisfaction rating of 4.46 out of 5.00.